Large Language Models: A Structured Taxonomy and Review of Challenges, Limitations, Solutions, and Future Directions
Source abstract
Abstract
Large language models (LLMs), as one of the most advanced achievements in the field of natural language processing (NLP), have made significant progress in areas such as natural language understanding and generation. However, attempts to achieve the widespread use of these models have met numerous challenges, encompassing technical, social, ethical, and legal aspects. This paper provides a comprehensive review of the various challenges associated with LLMs and analyzes the key issues related to these technologies. Among the challenges discussed are model interpretability, biases in data and model outcomes, ethical concerns regarding privacy and data security, and their high computational requirements. Furthermore, the paper examines how these challenges impact the applications of LLMs in fields such as healthcare, law, media, and education, emphasizing the importance of addressing these issues in the development and deployment of these models. Additionally, solutions for improving the robustness and control of models against biases and quality issues are proposed. Finally, the paper looks at the future of LLM research and the challenges that need to be addressed for the responsible and effective use of this technology. The goal of this paper is to provide a comprehensive analysis of the challenges and issues surrounding LLMs in order to enable the optimal and ethical use of these technologies in real-world applications.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Industrial Engineering, Faculty of Engineering, Khatam University, Tehran 1991633357, Iran
Department of Computer Sciences, Science and Research Branch, Islamic Azad University, Tehran 1477893855, Iran
Faculty of Economic Sciences, Lucian Blaga University of Sibiu, 550324 Sibiu, Romania
References (100+)
Show source-provided references
1. Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M.S., Bohg, J., Bosselut, A., and Brunskill, E. (2021). On the opportunities and risks of foundation models. arXiv.
2. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
3. OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., and Altman, S. (2023). Gpt-4 technical report. arXiv.
4. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv.
5. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., and Azhar, F. (2023). Llama: Open and efficient foundation language models. arXiv.
6. Role of chat gpt in public health10.1007/s10439-023-03172-7 ↗
7. Driess, D., Xia, F., Sajjadi, M.S.M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., and Yu, T. (2023). Palm-e: An embodied multimodal language model. arXiv.
8. A review on large language models: Architectures, applications, taxonomies, open issues and challenges10.1109/ACCESS.2024.3365742 ↗
9. Cavnar, W.B., and Trenkle, J.M. (1994, January 11–13). N-gram-based text categorization. Proceedings of the SDAIR-94, 3rd Annual Symposium on Document Analysis and Information Retrieval, Las Vegas, NV, USA.
10. Hidden Markov Models
11. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv.
12. Pennington, J., Socher, R., and Manning, C.D. (2014, January 25–29). Glove: Global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1162 ↗
13. Mikolov, T., Karafiát, M., Burget, L., Cernocký, J., and Khudanpur, S. (2010, January 26–30). Recurrent neural network based language model. Proceedings of the Interspeech, Chiba, Japan. No. 3.10.21437/Interspeech.2010-343 ↗
14. A review of recurrent neural networks: LSTM cells and network architectures10.1162/neco_a_01199 ↗
15. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., and Askell, A. (2020, January 6–12). Language models are few-shot learners. Proceedings of the Advances in Neural Information Processing Systems, Virtual.
16. Kosmyna, N., Hauptmann, E., Yuan, Y.T., Situ, J., Liao, X.-H., Beresnitzky, A.V., Braunstein, I., and Maes, P. (2025). Your brain on chatgpt: Accumulation of cognitive debt when using an ai assistant for essay writing task. arXiv.
17. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools10.1111/jels.12413 ↗
18. ChatGPT for good? On opportunities and challenges of large language models for education10.1016/j.lindif.2023.102274 ↗
19. Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., and Kasirzadeh, A. (2022, January 21–24). Taxonomy of risks posed by language models. Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, Seoul, Republic of Korea.10.1145/3531146.3533088 ↗
20. Li, Z., Shi, Y., Liu, Z., Yang, F., Liu, N., and Du, M. (2024). Quantifying multilingual performance of large language models across languages. arXiv.
21. A survey on large language models for recommendation10.1007/s11280-024-01291-2 ↗
22. Patil, R., and Gudivada, V. (2024). A review of current trends, techniques, and challenges in large language models (llms). Appl. Sci., 14.10.20944/preprints202402.0357.v1 ↗
23. He, H., and Su, W.J. (2024). A Law of Next-Token Prediction in Large Language Models. arXiv.
24. Bender, E.M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021, January 3–10). On the dangers of stochastic parrots: Can language models be too big?. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual.10.1145/3442188.3445922 ↗
25. Zhu, Y., Du, S., Li, B., Luo, Y., and Tang, N. (2024). Are Large Language Models Good Statisticians?. arXiv.
26. Almazrouei, E., Alobeidli, H., Alshamsi, A., Cappelli, A., Cojocaru, R., Debbah, M., Goffinet, É., Hesslow, D., Launay, J., and Malartic, Q. (2023). The falcon series of open language models. arXiv.
27. Naveed, H., Khan, A.U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., and Mian, A. (2023). A comprehensive overview of large language models. ACM Trans. Intell. Syst. Technol.
28. Xu, W., Hu, W., Wu, F., and Sengamedu, S. (2023). DeTiME: Diffusion-enhanced topic modeling using encoder-decoder based LLM. arXiv.10.18653/v1/2023.findings-emnlp.606 ↗
29. Transformer in transformer
30. Mo, Y., Qin, H., Dong, Y., Zhu, Z., and Li, Z. (2024). Large language model (llm) ai text generation detection based on transformer deep learning algorithm. arXiv.
31. The NLP cookbook: Modern recipes for transformer based deep learning architectures10.1109/ACCESS.2021.3077350 ↗
32. Li, T., El Mesbahi, Y., Kobyzev, I., Rashid, A., Mahmud, A., Anchuri, N., Hajimolahoseini, H., Liu, Y., and Rezagholizadeh, M. (2021). A short study on compressing decoder-based language models. arXiv.
33. Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. (2017). Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv.
34. A survey on evaluation of large language models10.1145/3641289 ↗
35. Palm: Scaling language modeling with pathways
36. Wu, Z., Qiu, L., Ross, A., Akyürek, E., Chen, B., Wang, B., Kim, N., Andreas, J., and Kim, Y. (2024, January 16–21). Reasoning or reciting? Exploring the capabilities and limitations of language models through counterfactual tasks. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Mexico City, Mexico.10.18653/v1/2024.naacl-long.102 ↗
37. Christiano, P.F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. (2017, January 4–9). Deep reinforcement learning from human preferences. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
38. Liu, Y., Cao, J., Liu, C., Ding, K., and Jin, L. (2024). Datasets for large language models: A comprehensive survey. arXiv.10.21203/rs.3.rs-3996137/v1 ↗
39. Lhoest, Q., del Moral, A.V., Jernite, Y., Thakur, A., von Platen, P., Patil, S., Chaumond, J., Drame, M., Plu, J., and Tunstall, L. (2021). Datasets: A community library for natural language processing. arXiv.10.18653/v1/2021.emnlp-demo.21 ↗
40. Song, Y., Cui, C., Khanuja, S., Liu, P., Faisal, F., Ostapenko, A., Indra Winata, G., Fikri Aji, A., Cahyawijaya, S., and Svetkov, Y. (2023). GlobalBench: A benchmark for global progress in natural language processing. arXiv.10.18653/v1/2023.emnlp-main.875 ↗
41. Kazemi, M., Dikkala, N., Anand, A., Devic, P., Dasgupta, I., Liu, F., Fatemi, B., Awasthi, P., Gollapudi, S., and Guo, D. (2024, January 10–15). Remi: A dataset for reasoning with multiple images. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada.
42. Pre-trained language models and their applications10.1016/j.eng.2022.04.024 ↗
43. Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., and Erlingsson, Ú. (2021, January 11–13). Extracting training data from large language models. Proceedings of the 30th USENIX Security Symposium (USENIX Security 21), Virtual.
44. Pre-trained models for natural language processing: A survey10.1007/s11431-020-1647-3 ↗
45. Que, H., Liu, J., Zhang, G., Zhang, C., Qu, X., Ma, Y., Duan, F., Bai, Z., Wang, J., and Zhang, Y. (2024, January 10–15). D-cpt law: Domain-specific continual pre-training scaling law for large language models. Proceedings of the Advances in Neural Information Processing Systems, Vancouver, BC, Canada.
46. Domain-specific language model pretraining for biomedical natural language processing
47. Large language models: A comprehensive survey of its applications, challenges, limitations, and future prospects
48. Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., and Carlini, N. (2021). Deduplicating training data makes language models better. arXiv.10.18653/v1/2022.acl-long.577 ↗
49. Zhang, S., Dong, L., Li, X., Zhang, S., Sun, X., Wang, S., Li, J., Hu, R., Zhang, T., and Wu, F. (2023). Instruction tuning for large language models: A survey. arXiv.
50. Honovich, O., Scialom, T., Levy, O., and Schick, T. (2022). Unnatural instructions: Tuning language models with (almost) no human labor. arXiv.10.18653/v1/2023.acl-long.806 ↗
51. Ahmad, W.U., Ficek, A., Samadi, M., Huang, J., Noroozi, V., Majumdar, S., and Ginsburg, B. (2025). OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs. arXiv.
52. Zhang, X., Tian, C., Yang, X., Chen, L., Li, Z., and Petzold, L.R. (2023). Alpacare: Instruction-tuned large language models for medical application. arXiv.
53. Cui, G., Yuan, L., Ding, N., Yao, G., He, B., Zhu, W., Ni, Y., Xie, G., Xie, R., and Lin, Y. (2024, January 21–27). Ultrafeedback: Boosting language models with high-quality feedback. Proceedings of the 41st International Conference on Machine Learning, Vienna, Austria.
54. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., and Ray, A. (December, January 28). Training language models to follow instructions with human feedback. Proceedings of the Advances in Neural Information Processing Systems, New Orleans, LA, USA.
55. Language model behavior: A comprehensive survey10.1162/coli_a_00492 ↗
56. Large language models in health care: Development, applications, and challenges10.1002/hcs2.61 ↗
57. Explainable AI (XAI) in healthcare: Enhancing trust and transparency in critical decision-making
58. Security and privacy challenges of large language models: A survey10.1145/3712001 ↗
59. Huang, J., and Chang, K.C.C. (2022). Towards reasoning in large language models: A survey. arXiv.10.18653/v1/2023.findings-acl.67 ↗
60. Lehman, E., Jain, S., Pichotta, K., Goldberg, Y., and Wallace, B.C. (2021). Does BERT pretrained on clinical notes reveal sensitive data?. arXiv.10.18653/v1/2021.naacl-main.73 ↗
61. Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 13–17). “Why should i trust you?” Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.10.1145/2939672.2939778 ↗
62. Large language models in medicine10.1038/s41591-023-02448-8 ↗
63. Challenges and strategies for wide-scale artificial intelligence (AI) deployment in healthcare practices: A perspective for healthcare organizations10.1016/j.artmed.2024.102861 ↗
64. Nazi, Z.A., and Peng, W. (2024). Large language models in healthcare and medical domain: A review. Informatics, 11.10.3390/informatics11030057 ↗
65. Disparities in dermatology AI performance on a diverse, curated clinical image set10.1126/sciadv.abq6147 ↗
66. Bias recognition and mitigation strategies in artificial intelligence healthcare applications10.1038/s41746-025-01503-7 ↗
67. Evaluating and addressing demographic disparities in medical large language models: A systematic review10.1186/s12939-025-02419-0 ↗
68. Large language models in medicine: The potentials and pitfalls: A narrative review10.7326/M23-2772 ↗
69. Nie, Y., Kong, Y., Dong, X., Mulvey, J.M., Poor, H.V., Wen, Q., and Zohren, S. (2024). A survey of large language models for financial applications: Progress, prospects and challenges. arXiv.
70. Rane, N., Choudhary, S., and Rane, J. (2023). Explainable Artificial Intelligence (XAI) approaches for transparency and accountability in financial decision-making. SSRN Electron. J.10.2139/ssrn.4640316 ↗
71. Lu, G., Guo, X., Zhang, R., Zhu, W., and Liu, J. (2025). BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMs. arXiv.
72. Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. (2019). Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv.
73. Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Patwary, M., Korthikanti, V.A., Vainbrand, D., Kashinkunti, P., Bernauer, J., and Catanzaro, B. (2021, January 14–19). Efficient large-scale language model training on gpu clusters using megatron-lm. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, St. Louis, MO, USA.10.1145/3458817.3476209 ↗
74. Araci, D. (2019). Finbert: Financial sentiment analysis with pre-trained language models. arXiv.
75. Strubell, E., Ganesh, A., and McCallum, A. (2020, January 7–12). Energy and policy considerations for modern deep learning research. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA. No. 09.10.1609/aaai.v34i09.7123 ↗
76. Li, Y., Wang, S., Ding, H., and Chen, H. (2023, January 27–29). Large language models in finance: A survey. Proceedings of the Fourth ACM International Conference on AI in Finance, New York, NY, USA.10.1145/3604237.3626869 ↗
77. Chu, Z., Guo, H., Zhou, X., Wang, Y., Yu, F., Chen, H., Xu, W., Lu, X., Cui, Q., and Li, L. (2023). Data-centric financial large language models. arXiv.
78. Phogat, K.S., Puranam, S.A., Dasaratha, S., Harsha, C., and Ramakrishna, S. (2024). Fine-tuning Smaller Language Models for Question Answering over Financial Documents. arXiv.10.18653/v1/2024.findings-emnlp.617 ↗
79. Qian, L., Zhou, W., Wang, Y., Peng, X., Yi, H., Zhao, Y., Huang, J., Xie, Q., and Nie, J.-Y. (2025). Fino1: On the transferability of reasoning enhanced llms to finance. arXiv.
80. Wu, S., Irsoy, O., Lu, S., Dabravolski, V., Dredze, M., Gehrmann, S., Kambadur, P., Rosenberg, D., and Mann, G. (2023). BloombergGPT: A large language model for finance. arXiv.
81. Wang, Y., Wang, Y., Liu, Y., Bao, R., Harimoto, K., and Sun, X. (2025, January 19–20). Proxy Tuning for Financial Sentiment Analysis: Overcoming Data Scarcity and Computational Barriers. Proceedings of the Joint Workshop of the 9th Financial Technology and Natural Language Processing (FinNLP), the 6th Financial Narrative Processing (FNP), and the 1st Workshop on Large Language Models for Finance and Legal (LLMFinLegal), Abu Dhabi, United Arab Emirates.
82. Wu, J., Gan, W., Chen, Z., Wan, S., and Yu, P.S. (2023, January 15–18). Multimodal large language models: A survey. Proceedings of the 2023 IEEE International Conference on Big Data (BigData), Sorrento, Italy.10.1109/BigData59044.2023.10386743 ↗
83. Mirishli, S. (2025). Regulating Ai In Financial Services: Legal Frameworks and Compliance Challenges. arXiv.
84. Rao, V., Sun, Y., Kumar, M., Mutneja, T., Mukherjee, A., and Yang, H. (2025). LLMs Meet Finance: Fine-Tuning Foundation Models for the Open FinLLM Leaderboard. arXiv.10.1109/IDS66066.2025.00019 ↗
85. Tavasoli, A., Sharbaf, M., and Madani, S.M. (2025). Responsible Innovation: A Strategic Framework for Financial LLM Integration. arXiv.
86. Huang, C., Nourian, A., and Griest, K. (2021). Hidden technical debts for fair machine learning in financial services. arXiv.
87. Large language models and sentiment analysis in financial markets: A review, datasets and case study10.1109/ACCESS.2024.3445413 ↗
88. Abdelsamie, M., and Wang, H. (2024, January 20–22). Comparative analysis of LLM-based market prediction and human expertise with sentiment analysis and machine learning integration. Proceedings of the 2024 7th International Conference on Data Science and Information Technology (DSIT), Nanjing, China.10.1109/DSIT61374.2024.10881868 ↗
89. ChatGPT: Unlocking the future of NLP in finance10.61351/mf.v1i1.43 ↗
90. Vidgof, M., Bachhofner, S., and Mendling, J. (2023, January 11–15). Large language models for business process management: Opportunities and challenges. Proceedings of the International Conference on Business Process Management, Utrecht, The Netherlands.10.1007/978-3-031-41623-1_7 ↗
91. Fahland, D., Fournier, F., Limonad, L., Skarbovsky, I., and Swevels, A.J. (2024). How well can large language models explain business processes?. arXiv.10.1016/j.datak.2025.102416 ↗
92. Nasseri, M., Brandtner, P., Zimmermann, R., Falatouri, T., Darbanian, F., and Obinwanne, T. (2023, January 23). Applications of large language models (LLMs) in business analytics–exemplary use cases in data preparation tasks. Proceedings of the International Conference on Human-Computer Interaction, Copenhagen, Denmark.10.1007/978-3-031-48057-7_12 ↗
93. Ferrara, E. (2023). Should chatgpt be biased? challenges and risks of bias in large language models. arXiv.10.2139/ssrn.4627814 ↗
94. Shen, S., Logeswaran, L., Lee, M., Lee, H., Poria, S., and Mihalcea, R. (2024). Understanding the capabilities and limitations of large language models for cultural commonsense. arXiv.10.18653/v1/2024.naacl-long.316 ↗
95. Advancements and applications of generative artificial intelligence and large language models on business management: A comprehensive review10.32996/jcsts.2024.6.1.26 ↗
96. Welcome to the era of chatgpt et al. the prospects of large language models10.1007/s12599-023-00795-x ↗
97. Raza, M., Jahangir, Z., Riaz, M.B., Saeed, M.J., and Sattar, M.A. (2025). Industrial applications of large language models. Sci. Rep., 15.10.1038/s41598-025-98483-1 ↗
98. Systems engineering issues for industry applications of large language model10.1016/j.asoc.2023.111165 ↗
99. Analyzing differences of highway lane-changing behavior using vehicle trajectory data10.1016/j.physa.2023.128980 ↗
100. Chen, X., Wei, C., Xin, Z., Zhao, J., and Xian, J. (2023). Ship detection under low-visibility weather interference via an ensemble generative adversarial network. J. Mar. Sci. Eng., 11.10.3390/jmse11112065 ↗