From Word Embeddings to Pre-Trained Language Models: A State-of-the-Art Walkthrough
Source abstract
Abstract
With the recent advances in deep learning, different approaches to improving pre-trained language models (PLMs) have been proposed. PLMs have advanced state-of-the-art (SOTA) performance on various natural language processing (NLP) tasks such as machine translation, text classification, question answering, text summarization, information retrieval, recommendation systems, named entity recognition, etc. In this paper, we provide a comprehensive review of prior embedding models as well as current breakthroughs in the field of PLMs. Then, we analyse and contrast the various models and provide an analysis of the way they have been built (number of parameters, compression techniques, etc.). Finally, we discuss the major issues and future directions for each of the main points.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Text Readability and Simplification
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Computers and Information Systems, Umm Al-Qura University, Mecca 24382, Saudi Arabia
Higher Institute of Computer Sciences and Mathematics, Monastir University, Monastir 5000, Tunisia
Funding
Deanship of Scientific Research at Umm Al-Qura University
References (90)
Show source-provided references
1. Distributional structure10.1080/00437956.1954.11659520 ↗
2. A Neural Probabilistic Language Model
3. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., and Dean, J. (2013, January 5–10). Distributed representations of words and phrases and their compositionality. Proceedings of the 26th International Conference on Neural Information Processing Systems, Lake Tahoe, NV, USA.
4. Pennington, J., Socher, R., and Manning, C.D. (2014, January 25–29). Glove: Global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP2014), Doha, Qatar.10.3115/v1/D14-1162 ↗
5. Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., and Weinberger, K.Q. (2014). Neural Word Embedding as Implicit Matrix Factorization. Advances in Neural Information Processing Systems 27, Proceedings of the 27th International Conference on Neural Information Processing Systems, Montreal, QC, Canada, 8–13 December 2014, MIT Press.
6. Liu, Q., Huang, H., Gao, Y., Wei, X., Tian, Y., and Liu, L. (2018, January 20–26). Task-oriented Word Embedding for Text Classification. Proceedings of the 27th International Conference on Computational Linguistics (COLING), Santa Fe, NM, USA.
7. Doan, T.M. (2018). Learning Word Embeddings. [Ph.D. Thesis, University Jean Monnet].
8. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv.
9. Liu, X., He, P., Chen, W., and Gao, J. (2019). Multi-Task Deep Neural Networks for Natural Language Understanding. arXiv.10.18653/v1/P19-1441 ↗
10. Language Models are Few-Shot Learners
11. Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., and Phang, J. (2022). GPT-NeoX-20B: An Open-Source Autoregressive Language Model. arXiv.10.18653/v1/2022.bigscience-1.9 ↗
12. Dumais, S.T., Furnas, G.W., Landauer, T.K., Deerwester, S., and Harshman, R. (1988, January 15–19). Using Latent Semantic Analysis to Improve Access to Textual Information. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Washington, DC, USA.10.1145/57167.57214 ↗
13. Indexing by latent semantic analysis10.1002/(SICI)1097-4571(199009)41:6<391::AID-ASI1>3.0.CO;2-9 ↗
14. Visualizing High-Dimensional Data Using t-SNE
15. Experiments on the application of IOHMMs to model financial returns series10.1109/72.896800 ↗
16. Collobert, R., and Weston, J. (2008, January 5–9). A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning. Proceedings of the 25th International Conference on Machine Learning (ICML ’08), Helsinki, Finland.10.1145/1390156.1390177 ↗
17. Enriching Word Vectors with Subword Information10.1162/tacl_a_00051 ↗
18. Bag of Tricks for Efficient Text Classification
19. McCann, B., Bradbury, J., Xiong, C., and Socher, R. (2017, January 4–9). Learned in Translation: Contextualized Word Vectors. Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.
20. Akbik, A., Blythe, D., and Vollgraf, R. (2018, January 20–26). Contextual String Embeddings for Sequence Labeling. Proceedings of the 27th International Conference on Computational Linguistics (COLING), Santa Fe, NM, USA.
21. Heinzerling, B., and Strube, M. (2018, January 7–12). BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan.
22. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention Is All You Need. arXiv.
23. Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018, January 1–6). Deep contextualized word representations. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2018), New Orleans, LA, USA.10.18653/v1/N18-1202 ↗
24. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. (2022, April 18). Improving Language Understanding by Generative Pre-Training. OpenAI. Available online: https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf.
25. Howard, J., and Ruder, S. (2018). Fine-tuned Language Models for Text Classification. arXiv.
26. Dai, A.M., and Le, Q.V. (2015). Semi-supervised Sequence Learning. arXiv.
27. Peters, M., Ammar, W., Bhagavatula, C., and Power, R. (August, January 30). Semi-supervised sequence tagging with bidirectional language models. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL 2017), Vancouver, BC, Canada.10.18653/v1/P17-1161 ↗
28. Universal Language Model Fine-tuning for Text Classification10.18653/v1/P18-1031 ↗
29. Zhu, Y., Kiros, R., Zemel, R.S., Salakhutdinov, R., Urtasun, R., Torralba, A., and Fidler, S. (2015, January 7–13). Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.11 ↗
30. Dai, Z., Yang, Z., Yang, Y., Carbonell, J.G., Le, Q.V., and Salakhutdinov, R. (2019). Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context. arXiv.10.18653/v1/P19-1285 ↗
31. Language Models are Unsupervised Multitask Learners
32. Sun, Y., Wang, S., Li, Y., Feng, S., Chen, X., Zhang, H., Tian, X., Zhu, D., Tian, H., and Wu, H. (2019). ERNIE: Enhanced Representation through Knowledge Integration. arXiv.
33. Yang, Z., Dai, Z., Yang, Y., Carbonell, J.G., Salakhutdinov, R., and Le, Q.V. (2019). XLNet: Generalized Autoregressive Pretraining for Language Understanding. arXiv.
34. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv.
35. Lample, G., and Conneau, A. (2019). Cross-lingual Language Model Pretraining. arXiv.
36. Keskar, N.S., McCann, B., Varshney, L.R., Xiong, C., and Socher, R. (2019). CTRL: A Conditional Transformer Language Model for Controllable Generation. arXiv.
37. Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. (2019). Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv.
38. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (2019). ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. arXiv.
39. Bucilua Cristian, C., Caruana, R., and Niculescu-Mizil, A. (2006). Model Compression. KDD ’06, Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Philadelphia, PA, USA, 20–23 August 2006, Association for Computing Machinery.10.1145/1150402.1150464 ↗
40. Hinton, G.E., Vinyals, O., and Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv.
41. Sanh, V., Debut, L., Chaumond, J., and Wolf, T. (2019). DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv.
42. Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., and Liu, Q. (2019). TinyBERT: Distilling BERT for Natural Language Understanding. arXiv.10.18653/v1/2020.findings-emnlp.372 ↗
43. Zhao, S., Gupta, R., Song, Y., and Zhou, D. (2020, January 26–30). Extreme language model compression with optimal subwords and shared projections. Proceedings of the 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia.
44. Sun, Z., Yu, H., Song, X., Liu, R., Yang, Y., and Zhou, D. (2020, January 5–10). Mobilebert: Task-agnostic compression of bert for resource limited devices. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.195 ↗
45. Tsai, H., Riesa, J., Johnson, M., Arivazhagan, N., Li, X., and Archer, A. (2019). Small and practical BERT models for sequence labeling. arXiv.10.18653/v1/D19-1374 ↗
46. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P.J. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv.
47. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2019). BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. arXiv.10.18653/v1/2020.acl-main.703 ↗
48. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. (2019). Unsupervised Cross-lingual Representation Learning at Scale. arXiv.10.18653/v1/2020.acl-main.747 ↗
49. PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
50. Kitaev, N., Kaiser, L., and Levskaya, A. (2020). Reformer: The Efficient Transformer. arXiv.
51. Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M. (2020). REALM: Retrieval-Augmented Language Model Pre-Training. arXiv.
52. Rosset, C. (2022, April 19). Turing-NLG: A 17-Billion-Parameter Language Model by Microsoft. Microsoft Blog. Available online: https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/.
53. Clark, K., Luong, M., Le, Q.V., and Manning, C.D. (2020). ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. arXiv.
54. Herzig, J., Nowak, P.K., Müller, T., Piccinno, F., and Eisenschlos, J.M. (2020). TAPAS: Weakly Supervised Table Parsing via Pre-training. arXiv.10.18653/v1/2020.acl-main.398 ↗
55. Beltagy, I., Peters, M.E., and Cohan, A. (2020). Longformer: The Long-Document Transformer. arXiv.
56. MPNet: Masked and Permuted Pre-training for Language Understanding
57. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
58. He, P., Liu, X., Gao, J., and Chen, W. (2020). DeBERTa: Decoding-enhanced BERT with Disentangled Attention. arXiv.
59. Pre-training via Paraphrasing
60. Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang, Y., Krikun, M., Shazeer, N., and Chen, Z. (2021, January 3–7). {GS}hard: Scaling Giant Models with Conditional Computation and Automatic Sharding. Proceedings of the International Conference on Learning Representations, Virtual Event.
61. Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. (2020). mT5: A massively multilingual pre-trained text-to-text transformer. arXiv.10.18653/v1/2021.naacl-main.41 ↗
62. KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation10.1162/tacl_a_00360 ↗
63. Chi, Z., Huang, S., Dong, L., Ma, S., Singhal, S., Bajaj, P., Song, X., and Wei, F. (2021). XLM-E: Cross-lingual Language Model Pre-training via ELECTRA. arXiv.10.18653/v1/2022.acl-long.427 ↗
64. Smith, S., Patwary, M., Norick, B., LeGresley, P., Rajbhandari, S., Casper, J., Liu, Z., Prabhumoye, S., Zerveas, G., and Korthikanti, V. (2022). Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model. arXiv.
65. Sanh, V., Webson, A., Raffel, C., Bach, S.H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T.L., and Raja, A. (2021). Multitask Prompted Training Enables Zero-Shot Task Generalization. arXiv.
66. Edalati, A., Tahaei, M.S., Rashid, A., Nia, V.P., Clark, J.J., and Rezagholizadeh, M. (2021). Kronecker Decomposition for GPT Compression. arXiv.10.18653/v1/2022.acl-short.24 ↗
67. Du, N., Huang, Y., Dai, A.M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A.W., and Firat, O. (2021). GLaM: Efficient Scaling of Language Models with Mixture-of-Experts. arXiv.
68. Rae, J.W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, H.F., Aslanides, J., Henderson, S., Ring, R., and Young, S. (2021). Scaling Language Models: Methods, Analysis & Insights from Training Gopher. arXiv.
69. Lin, X.V., Mihaylov, T., Artetxe, M., Wang, T., Chen, S., Simig, D., Ott, M., Goyal, N., Bhosale, S., and Du, J. (2021). Few-shot Learning with Multilingual Language Models. arXiv.10.18653/v1/2022.emnlp-main.616 ↗
70. Thoppilan, R., Freitas, D.D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H., Jin, A., Bos, T., Baker, L., and Du, Y. (2022). LaMDA: Language Models for Dialog Applications. arXiv.
71. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., and Nabeshima, N. (2021). The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv.
72. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D.d.L., Hendricks, L.A., Welbl, J., and Clark, A. (2022). Training Compute-Optimal Large Language Models. arXiv.
73. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H.W., Sutton, C., and Gehrmann, S. (2022). PaLM: Scaling Language Modeling with Pathways. arXiv.
74. Pathways: Asynchronous Distributed Dataflow for ML
75. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., and Lin, X.V. (2022). OPT: Open Pre-trained Transformer Language Models. arXiv.
76. Karande, H., Walambe, R., Benjamin, V., Kotecha, K., and Raghu, T.S. (2021). Stance Detection with BERT Embeddings for Credibility Analysis of Information on Social Media. PeerJ Comput. Sci., 7.10.7717/peerj-cs.467 ↗
77. A Deep Learning Model Based on BERT and Sentence Transformer for Semantic Keyphrase Extraction on Big Social Data10.1109/ACCESS.2021.3133651 ↗
78. Sun, Y., Zheng, Y., Hao, C., and Qiu, H. (2021). NSP-BERT: A Prompt-based Zero-Shot Learner Through an Original Pre-training Task-Next Sentence Prediction. arXiv.
79. Xu, H., Chen, Y., Du, Y., Shao, N., Wang, Y., Li, H., and Yang, Z. (2022). ZeroPrompt: Scaling Prompt-Based Pretraining to 1000 Tasks Improves Zero-Shot Generalization. arXiv.10.18653/v1/2022.findings-emnlp.312 ↗
80. Frankle, J., and Carbin, M. (2018). The Lottery Ticket Hypothesis: Training Pruned Neural Networks. arXiv.
81. Liu, D., Cheng, P., Dong, Z., He, X., Pan, W., and Ming, Z. (2020, January 25–30). A General Knowledge Distillation Framework for Counterfactual Recommendation via Uniform Data. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, Online.10.1145/3397271.3401083 ↗
82. Ding, F., Luo, F., Hu, H., and Yang, Y. (2020). Multi-level Knowledge Distillation. arXiv.
83. Sun, S., Cheng, Y., Gan, Z., and Liu, J. (2019). Patient Knowledge Distillation for BERT Model Compression. arXiv.10.18653/v1/D19-1441 ↗
84. Wang, T., Zhu, J., Torralba, A., and Efros, A.A. (2018). Dataset Distillation. arXiv.
85. Chumachenko, A., Gavrilov, D., Balagansky, N., and Kalaidin, P. (2020). Weight squeezing: Reparameterization for extreme compression and fast inference. arXiv.
86. Compressing Large-Scale Transformer-Based Models: A Case Study on BERT10.1162/tacl_a_00413 ↗
87. Compression of Deep Learning Models for Text: A Survey
88. Green AI10.1145/3381831 ↗
89. Verdecchia, R., Cruz, L., Sallou, J., Lin, M., Wickenden, J., and Hotellier, E. (2022). Data-Centric Green AI: An Exploratory Empirical Study. arXiv.10.1109/ICT4S55073.2022.00015 ↗
90. Strubell, E., Ganesh, A., and McCallum, A. (2019). Energy and Policy Considerations for Deep Learning in NLP. arXiv.10.18653/v1/P19-1355 ↗