A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)
Source abstract
Abstract
Natural language processing (NLP) has significantly transformed in the last decade, especially in the field of language modeling. Large language models (LLMs) have achieved SOTA performances on natural language understanding (NLU) and natural language generation (NLG) tasks by learning language representation in self-supervised ways. This paper provides a comprehensive survey to capture the progression of advances in language models. In this paper, we examine the different aspects of language models, which started with a few million parameters but have reached the size of a trillion in a very short time. We also look at how these LLMs transitioned from task-specific to task-independent to task-and-language-independent architectures. This paper extensively discusses different pretraining objectives, benchmarks, and transfer learning methods used in LLMs. It also examines different finetuning and in-context learning techniques used in downstream tasks. Moreover, it explores how LLMs can perform well across many domains and datasets if sufficiently trained on a large and diverse dataset. Next, it discusses how, over time, the availability of cheap computational power and large datasets have improved LLM’s capabilities and raised new challenges. As part of our study, we also inspect LLMs from the perspective of scalability to see how their performance is affected by the model’s depth, width, and data size. Lastly, we provide an empirical comparison of existing trends and techniques and a comprehensive analysis of where the field of LLM currently stands.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Data Quality and Management
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computing, Grand Valley State University, Allendale Charter Township, MI 49401, USA
Computer Science Department, East Carolina University, Greenville, NC 27858, USA
References (100+)
Show source-provided references
1. Distributional structure10.1080/00437956.1954.11659520 ↗
2. A statistical approach to machine translation
3. Computer evaluation of indexing and text processing10.1145/321439.321441 ↗
4. A statistical interpretation of term specificity and its application in retrieval10.1108/eb026526 ↗
5. A vector space model for automatic indexing10.1145/361219.361220 ↗
6. Tang, B., Shepherd, M., Milios, E., and Heywood, M.I. (2005, January 21). Comparing and combining dimension reduction techniques for efficient text clustering. Proceedings of the SIAM International Workshop on Feature Selection for Data Mining, Newport Beach, CA, USA.
7. Independent component analysis: Algorithms and applications10.1016/S0893-6080(00)00026-5 ↗
8. Le, Q., and Mikolov, T. (2014, January 8–13). Distributed representations of sentences and documents. Proceedings of the International Conference on Machine Learning, Montreal, QC, Canada.
9. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv.
10. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., and Dean, J. (2013). Distributed representations of words and phrases and their compositionality. Adv. Neural Inf. Process. Syst., 26.
11. Pennington, J., Socher, R., and Manning, C.D. (2014, January 25–29). Glove: Global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1162 ↗
12. Enriching word vectors with subword information10.1162/tacl_a_00051 ↗
13. Vilnis, L., and McCallum, A. (2014). Word representations via gaussian embedding. arXiv.
14. Athiwaratkun, B., and Wilson, A.G. (2017). Multimodal word distributions. arXiv.10.18653/v1/P17-1151 ↗
15. Melamud, O., Goldberger, J., and Dagan, I. (2016, January 11–12). context2vec: Learning generic context embedding with bidirectional lstm. Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, Berlin, Germany.10.18653/v1/K16-1006 ↗
16. McCann, B., Bradbury, J., Xiong, C., and Socher, R. (2017, January 4–9). Learned in translation: Contextualized word vectors. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA.
17. Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018, January 1–6). Deep Contextualized Word Representations. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, LA, USA.10.18653/v1/N18-1202 ↗
18. Howard, J., and Ruder, S. (2018). Universal language model fine-tuning for text classification. arXiv.10.18653/v1/P18-1031 ↗
19. Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., and Hon, H.W. (2019). Unified language model pre-training for natural language understanding and generation. Adv. Neural Inf. Process. Syst., 32.
20. Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv.10.3115/v1/D14-1179 ↗
21. Sutskever, I., Vinyals, O., and Le, Q.V. (2014). Sequence to sequence learning with neural networks. Adv. Neural Inf. Process. Syst., 27.
22. Scao, T.L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A.S., Yvon, F., and Gallé, M. (2022). Bloom: A 176b-parameter open-access multilingual language model. arXiv.
23. Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv.
24. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Adv. Neural Inf. Process. Syst., 30.
25. Exploring the limits of transfer learning with a unified text-to-text transformer
26. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
27. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. (2024, February 01). Improving Language Understanding by Generative Pre-Training. Available online: https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf.
28. Language models are unsupervised multitask learners
29. Language models are few-shot learners
30. Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M. (2020, January 9–11). Retrieval augmented language model pre-training. Proceedings of the International Conference on Machine Learning, Online.
31. Jurassic-1: Technical details and evaluation
32. Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. (2020). mT5: A massively multilingual pre-trained text-to-text transformer. arXiv.10.18653/v1/2021.naacl-main.41 ↗
33. Zeng, W., Ren, X., Su, T., Wang, H., Liao, Y., Wang, Z., Jiang, X., Yang, Z., Wang, K., and Zhang, X. (2021). Pangu-α: Large-scale autoregressive pretrained Chinese language models with auto-parallel computation. arXiv.
34. Cpm-2: Large-scale cost-effective pre-trained language models10.1016/j.aiopen.2021.12.003 ↗
35. Wu, S., Zhao, X., Yu, T., Zhang, R., Shen, C., Liu, H., Li, F., Zhu, H., Luo, J., and Xu, L. (2021). Yuan 1.0: Large-scale pre-trained language model in zero-shot and few-shot learning. arXiv.
36. Kim, B., Kim, H., Lee, S.W., Lee, G., Kwak, D., Jeon, D.H., Park, S., Kim, S., Kim, S., and Seo, D. (2021). What changes can large-scale language models bring? intensive study on hyperclova: Billions-scale korean generative pretrained transformers. arXiv.10.18653/v1/2021.emnlp-main.274 ↗
37. Du, N., Huang, Y., Dai, A.M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A.W., and Firat, O. (2022, January 17–23). Glam: Efficient scaling of language models with mixture-of-experts. Proceedings of the International Conference on Machine Learning, Baltimore, MD, USA.
38. Sun, Y., Wang, S., Feng, S., Ding, S., Pang, C., Shang, J., Liu, J., Chen, X., Zhao, Y., and Lu, Y. (2021). Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. arXiv.
39. Rae, J.W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., and Young, S. (2021). Scaling language models: Methods, analysis & insights from training gopher. arXiv.
40. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D.D.L., Hendricks, L.A., Welbl, J., and Clark, A. (2022). Training compute-optimal large language models. arXiv.
41. Competition-level code generation with alphacode10.1126/science.abq1158 ↗
42. Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., and Xiong, C. (2022). Codegen: An open large language model for code with multi-turn program synthesis. arXiv.
43. Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Wang, Z., Shen, L., Wang, A., and Li, Y. (2023). Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv.10.1145/3580305.3599790 ↗
44. Wei, J., Bosma, M., Zhao, V.Y., Guu, K., Yu, A.W., Lester, B., Du, N., Dai, A.M., and Le, Q.V. (2021). Finetuned language models are zero-shot learners. arXiv.
45. Training language models to follow instructions with human feedback
46. Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.T., Jin, A., Bos, T., Baker, L., and Du, Y. (2022). Lamda: Language models for dialog applications. arXiv.
47. Sanh, V., Webson, A., Raffel, C., Bach, S.H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T.L., and Raja, A. (2021). Multitask prompted training enables zero-shot task generalization. arXiv.
48. Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., and Phang, J. (2022). Gpt-neox-20b: An open-source autoregressive language model. arXiv.10.18653/v1/2022.bigscience-1.9 ↗
49. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., and Lin, X.V. (2022). Opt: Open pre-trained transformer language models. arXiv.
50. Solving quantitative reasoning problems with language models
51. Soltan, S., Ananthakrishnan, S., FitzGerald, J., Gupta, R., Hamza, W., Khan, H., Peris, C., Rawls, S., Rosenbaum, A., and Rumshisky, A. (2022). Alexatm 20b: Few-shot learning using a large-scale multilingual seq2seq model. arXiv.
52. Zeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., and Xia, X. (2022). Glm-130b: An open bilingual pre-trained model. arXiv.
53. Lin, X.V., Mihaylov, T., Artetxe, M., Wang, T., Chen, S., Simig, D., Ott, M., Goyal, N., Bhosale, S., and Du, J. (2022, January 7–11). Few-shot Learning with Multilingual Generative Language Models. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates.10.18653/v1/2022.emnlp-main.616 ↗
54. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H.W., Sutton, C., and Gehrmann, S. (2022). Palm: Scaling language modeling with pathways. arXiv.
55. Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., and Stojnic, R. (2022). Galactica: A large language model for science. arXiv.
56. Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A.J., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., and Beyer, L. (2022). Pali: A jointly-scaled multilingual language-image model. arXiv.
57. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., and Azhar, F. (2023). Llama: Open and efficient foundation language models. arXiv.
58. Tay, Y., Dehghani, M., Tran, V.Q., Garcia, X., Wei, J., Wang, X., Chung, H.W., Bahri, D., Schuster, T., and Zheng, S. (2022, January 25–29). Ul2: Unifying language learning paradigms. Proceedings of the Eleventh International Conference on Learning Representations, Virtual Event.
59. Biderman, S., Schoelkopf, H., Anthony, Q.G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M.A., Purohit, S., Prashanth, U.S., and Raff, E. (2023, January 10–12). Pythia: A suite for analyzing large language models across training and scaling. Proceedings of the International Conference on Machine Learning, Nashville, TN, USA.
60. Su, H., Zhou, X., Yu, H., Chen, Y., Zhu, Z., Yu, Y., and Zhou, J. (2022). Welm: A well-read pre-trained language model for chinese. arXiv.
61. Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., and Tang, J. (2021). Glm: General language model pretraining with autoregressive blank infilling. arXiv.10.18653/v1/2022.acl-long.26 ↗
62. Tay, Y., Dehghani, M., Rao, J., Fedus, W., Abnar, S., Chung, H.W., Narang, S., Yogatama, D., Vaswani, A., and Metzler, D. (2021). Scale efficiently: Insights from pre-training and fine-tuning transformers. arXiv.
63. Artetxe, M., Bhosale, S., Goyal, N., Mihaylov, T., Ott, M., Shleifer, S., Lin, X.V., Du, J., Iyer, S., and Pasunuru, R. (2021). Efficient large scale language modeling with mixtures of experts. arXiv.10.18653/v1/2022.emnlp-main.804 ↗
64. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. (2019). Unsupervised cross-lingual representation learning at scale. arXiv.10.18653/v1/2020.acl-main.747 ↗
65. Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. (2017). Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv.
66. Zoph, B., Bello, I., Kumar, S., Du, N., Huang, Y., Dean, J., Shazeer, N., and Fedus, W. (2022). St-moe: Designing stable and transferable sparse expert models. arXiv.
67. Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. (2020). GShard: Scaling giant models with conditional computation and automatic sharding. arXiv.
68. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity
69. Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., and Liu, Q. (2019). ERNIE: Enhanced language representation with informative entities. arXiv.10.18653/v1/P19-1139 ↗
70. Peters, M.E., Neumann, M., Logan, R.L., Schwartz, R., Joshi, V., Singh, S., and Smith, N.A. (2019). Knowledge enhanced contextual word representations. arXiv.10.18653/v1/D19-1005 ↗
71. Zhou, W., Lee, D.H., Selvam, R.K., Lee, S., Lin, B.Y., and Ren, X. (2020). Pre-training text-to-text transformers for concept-centric common sense. arXiv.
72. Xiong, W., Du, J., Wang, W.Y., and Stoyanov, V. (2019). Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model. arXiv.
73. KEPLER: A unified model for knowledge embedding and pre-trained language representation10.1162/tacl_a_00360 ↗
74. Sun, T., Shao, Y., Qiu, X., Guo, Q., Hu, Y., Huang, X., and Zhang, Z. (2020). Colake: Contextualized language and knowledge embedding. arXiv.10.18653/v1/2020.coling-main.327 ↗
75. Wang, R., Tang, D., Duan, N., Wei, Z., Huang, X., Cao, G., Jiang, D., and Zhou, M. (2020). K-adapter: Infusing knowledge into pre-trained models with adapters. arXiv.10.18653/v1/2021.findings-acl.121 ↗
76. Tay, Y., Wei, J., Chung, H.W., Tran, V.Q., So, D.R., Shakeri, S., Garcia, X., Zheng, H.S., Rao, J., and Chowdhery, A. (2022). Transcending scaling laws with 0.1% extra compute. arXiv.10.18653/v1/2023.emnlp-main.91 ↗
77. Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. (2019, January 13). Parameter-efficient transfer learning for NLP. Proceedings of the International Conference on Machine Learning, Vancouver, BC, Canada.
78. Li, X.L., and Liang, P. (2021). Prefix-tuning: Optimizing continuous prompts for generation. arXiv.10.18653/v1/2021.acl-long.353 ↗
79. Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., and Tang, J. (2023). GPT understands, too. AI Open, in press.10.1016/j.aiopen.2023.08.012 ↗
80. Lester, B., Al-Rfou, R., and Constant, N. (2021). The power of scale for parameter-efficient prompt tuning. arXiv.10.18653/v1/2021.emnlp-main.243 ↗
81. Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Biderman, S., Scao, T.L., Bari, M.S., Shen, S., Yong, Z.X., and Schoelkopf, H. (2022). Crosslingual generalization through multitask finetuning. arXiv.10.18653/v1/2023.acl-long.891 ↗
82. Ziegler, D.M., Stiennon, N., Wu, J., Brown, T.B., Radford, A., Amodei, D., Christiano, P., and Irving, G. (2019). Fine-tuning language models from human preferences. arXiv.
83. Wu, J., Ouyang, L., Ziegler, D.M., Stiennon, N., Lowe, R., Leike, J., and Christiano, P. (2021). Recursively summarizing books with human feedback. arXiv.
84. Learning to summarize with human feedback
85. Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Arunkumar, A., Ashok, A., Dhanasekaran, A.S., Naik, A., and Stap, D. (2022). Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. arXiv.10.18653/v1/2022.emnlp-main.340 ↗
86. Iyer, S., Lin, X.V., Pasunuru, R., Mihaylov, T., Simig, D., Yu, P., Shuster, K., Wang, T., Liu, Q., and Koura, P.S. (2022). Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv.
87. Chung, H.W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., and Brahma, S. (2022). Scaling instruction-finetuned language models. arXiv.
88. Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N.A., Khashabi, D., and Hajishirzi, H. (2022). Self-instruct: Aligning language model with self generated instructions. arXiv.10.18653/v1/2023.acl-long.754 ↗
89. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.D.O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., and Brockman, G. (2021). Evaluating large language models trained on code. arXiv.
90. Zhao, Z., Wallace, E., Feng, S., Klein, D., and Singh, S. (2021, January 18–24). Calibrate before use: Improving few-shot performance of language models. Proceedings of the International Conference on Machine Learning, Virtual.
91. Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., and Chen, W. (2021). What Makes Good In-Context Examples for GPT-3?. arXiv.10.18653/v1/2022.deelio-1.10 ↗
92. Mosbach, M., Pimentel, T., Ravfogel, S., Klakow, D., and Elazar, Y. (2023). Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation. arXiv.10.18653/v1/2023.findings-acl.779 ↗
93. Wang, T., Roberts, A., Hesslow, D., Le Scao, T., Chung, H.W., Beltagy, I., Launay, J., and Raffel, C. (2022, January 28–30). What language model architecture and pretraining objective works best for zero-shot generalization?. Proceedings of the International Conference on Machine Learning, PMLR, Online.
94. Chain-of-thought prompting elicits reasoning in large language models
95. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. (2022). Self-consistency improves chain of thought reasoning in language models. arXiv.
96. Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., and Metzler, D. (2022). Emergent abilities of large language models. arXiv.
97. Kaplan, J., McCandlish, S., Henighan, T., Brown, T.B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling laws for neural language models. arXiv.
98. Limits to depth efficiencies of self-attention
99. Gehman, S., Gururangan, S., Sap, M., Choi, Y., and Smith, N.A. (2020). Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv.10.18653/v1/2020.findings-emnlp.301 ↗
100. Ung, M., Xu, J., and Boureau, Y.L. (2021). Saferdialogues: Taking feedback gracefully after conversational safety failures. arXiv.10.18653/v1/2022.acl-long.447 ↗