Pre-trained models for natural language processing: A survey
Source abstract
Abstract
Recently, the emergence of pre-trained models (PTMs) has brought natural language processing (NLP) to a new era. In this survey, we provide a comprehensive review of PTMs for NLP. We first briefly introduce language representation learning and its research progress. Then we systematically categorize existing PTMs based on a taxonomy with four perspectives. Next, we describe how to adapt the knowledge of PTMs to the downstream tasks. Finally, we outline some potential directions of PTMs for future research. This survey is purposed to be a hands-on guide for understanding, using, and developing PTMs for various NLP tasks.
Source: openaire · Quality: candidate unverified · Source record ↗
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Multimodal Machine Learning Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
References (100+)
Show source-provided references
1. Kalchbrenner N, Grefenstette E, Blunsom P. A convolutional neural network for modelling sentences. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics. Baltimore, 2014. 655–66510.3115/v1/P14-1062 ↗
2. Kim Y. Convolutional neural networks for sentence classification. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. Doha, 2014. 1746–175110.3115/v1/D14-1181 ↗
3. Gehring J, Auli M, Grangier D, et al. Convolutional sequence to sequence learning. In: Proceedings of the International Conference on Machine Learning. Sydney, 2017. 1243–1252
4. Sutskever I, Vinyals O, Le Q V. Sequence to sequence learning with neural networks. In: Proceedings of the Advances in Neural Information Processing Systems. Montreal, 2014. 3104–3112
5. Liu P, Qiu X, Huang X. Recurrent neural network for text classification with multi-task learning. In: Proceedings of the International Joint Conference on Artificial Intelligence. New York, 2016. 2873–2879
6. Socher R, Perelygin A, Wu J Y, et al. Recursive deep models for semantic compositionality over a sentiment treebank. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. Seattle, 2013. 1631–1642
7. Tai K S, Socher R, Manning C D. Improved semantic representations from tree-structured long short-term memory networks. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics. Beijing, 2015. 1556–156610.3115/v1/P15-1150 ↗
8. Marcheggiani D, Bastings J, Titov I. Exploiting semantics in neural machine translation with graph convolutional networks. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. New Orleans, 2018. 486–49210.18653/v1/N18-2078 ↗
9. Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to align and translate. In: Proceedings of the International Conference on Learning Representations. San Diego, 2015
10. Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. In: Proceedings of the Advances in Neural Information Processing Systems. Long Beach, 2017. 5998–6008
11. Mikolov T, Sutskever I, Chen K, et al. Distributed representations of words and phrases and their compositionality. In: Proceedings of the Advances in Neural Information Processing Systems. Lake Tahoe, 2013. 3111–3119
12. Pennington J, Socher R, Manning C D. GloVe: Global vectors for word representation. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. Doha, 2014. 1532–154310.3115/v1/D14-1162 ↗
13. McCann B, Bradbury J, Xiong C, et al. Learned in translation: Contextualized word vectors. In: Proceedings of the Advances in Neural Information Processing Systems. Long Beach, 2017. 6294–6305
14. Peters M E, Neumann M, Iyyer M, et al. Deep contextualized word representations. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. New Orleans, 2018. 2227–223710.18653/v1/N18-1202 ↗
15. Radford A, Narasimhan K, Salimans T, et al. Improving language understanding by generative pre-training. OpenAI Blog. 2018
16. Devlin J, Chang M, Lee K, et al. BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Minneapolis, 2019. 4171–4186
17. Bengio Y, Courville A, Vincent P. Representation learning: A review and new perspectives. IEEE Trans Pattern Anal Mach Intell, 2013, 35: 1798–182810.1109/TPAMI.2013.50 ↗
18. Kim Y, Jernite Y, Sontag D, et al. Character-aware neural language models. In: Proceedings of the AAAI Conference on Artificial Intelligence. Phoenix, 2016. 2741–274910.1609/aaai.v30i1.10362 ↗
19. Bojanowski P, Grave E, Joulin A, et al. Enriching word vectors with subword information. Trans Associat Comput Linguist, 2017, 5: 135–14610.1162/tacl_a_00051 ↗
20. Sennrich R, Haddow B, Birch A. Neural machine translation of rare words with subword units. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics. Berlin, 201610.18653/v1/P16-1162 ↗
21. Hochreiter S, Schmidhuber J. Long short-term memory. Neural Computation. 1997. 9: 1735–178010.1162/neco.1997.9.8.1735 ↗
22. Chung J, Gulcehre C, Cho K, et al. Empirical evaluation of gated recurrent neural networks on sequence modeling. ArXiv: 1412.3555
23. Zhu X, Sobihani P, Guo H. Long short-term memory over recursive structures. In: Proceedings of the International Conference on Machine Learning. Lille, 2015. 1604–1612
24. Kipf T N and Welling M. Semi-supervised classification with graph convolutional networks. In: Proceedings of the International Conference on Learning Representations. Toulon, 2017
25. Guo Q, Qiu X, Liu P, et al. Star-transformer. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Minneapolis, 2019. 1315–1325
26. Erhan D, Bengio Y, Courville A C, et al. Why does unsupervised pre-training help deep learning? J Mach Learn Res, 2010, 11: 625–660
27. Hinton G E. Reducing the dimensionality of data with neural networks. Science, 2006, 313: 504–50710.1126/science.1127647 ↗
28. Hinton G, McClelland J, Rumelhart D. Distributed representations. The Philosophy of Artificial Intelligence, 1990, 248–280
29. Bengio Y, Ducharme R, Vincent P, et al. A neural probabilistic language model. J Mach Learn Res, 2003, 3: 1137–1155
30. Collobert R, Weston J, Bottou L, et al. Natural language processing (almost) from scratch. J Mach Learn Res, 2011, 12: 2493–2537
31. Le Q, Mikolov T. Distributed representations of sentences and documents. In: Proceedings of the International Conference on Machine Learning. Beijing, 2014. 1188–1196
32. Kiros R, Zhu Y, Salakhutdinov R R, et al. Skip-thought vectors. In: Proceedings of the Advances in Neural Information Processing Systems. Montreal, 2015. 3294–3302
33. Melamud O, Goldberger J, Dagan I. Context2Vec: Learning generic context embedding with bidirectional LSTM. In: Proceedings of the Conference on Computational Natural Language Learning. Berlin, 2016. 51–6110.18653/v1/K16-1006 ↗
34. Dai A M and Le Q V. Semi-supervised sequence learning. In: Proceedings of the Advances in Neural Information Processing Systems. Montreal, 2015. 3079–3087
35. Ramachandran P, Liu P J, Le Q. Unsupervised pretraining for sequence to sequence learning. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. Copenhagen, 2017. 383–39110.18653/v1/D17-1039 ↗
36. Akbik A, Blythe D, Vollgraf R. Contextual string embeddings for sequence labeling. In: Proceedings of the International Conference on Computational Linguistics. Santa Fe, 2018. 1638–1649
37. Howard J and Ruder S. Universal language model fine-tuning for text classification. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics. Melbourne, 2018. 328–33910.18653/v1/P18-1031 ↗
38. Baevski A, Edunov S, Liu Y, et al. Cloze-driven pretraining of self-attention networks. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. Hong Kong, 2019. 5359–536810.18653/v1/D19-1539 ↗
39. Taylor W L. “Cloze Procedure”: A new tool for measuring readability. Jism Q, 1953, 30: 415–433
40. Song K, Tan X, Qin T, et al. MASS: Masked sequence to sequence pre-training for language generation. In: Proceedings of the International Conference on Machine Learning. Long Beach, 2019. 5926–5936
41. Raffel C, Shazeer N, Roberts A, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv: 1910.10683
42. Liu Y, Ott M, Goyal N, et al. RoBERTa: A robustly optimized BERT pretraining approach. ArXiv: 1907.11692
43. Dong L, Yang N, Wang W, et al. Unified language model pre-training for natural language understanding and generation. In: Proceedings of the Advances in Neural Information Processing Systems. Vancouver, 2019. 13042–13054
44. Bao H, Dong L, Wei F, et al. UniLMv2: Pseudo-masked language models for unified language model pre-training. ArXiv: 2002.12804
45. Conneau A, Lample G. Cross-lingual language model pretraining. In: Proceedings of the Advances in Neural Information Processing Systems. Vancouver, 2019. 7057–7067
46. Joshi M, Chen D, Liu Y, et al. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 2019, 8: 64–7710.1162/tacl_a_00300 ↗
47. Wang W, Bi B, Yan M, et al. StructBERT: Incorporating language structures into pre-training for deep language understanding. In: Proceedings of the International Conference on Learning Representations. Addis Ababa, 2020.
48. Yang Z, Dai Z, Yang Y, et al. XLNet: Generalized autoregressive pretraining for language understanding. In: Proceedings of the Advances in Neural Information Processing Systems. Vancouver, 2019. 5754–5764
49. Lewis M, Liu Y, Goyal N, et al. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. ArXiv: 1910.13461
50. Saunshi N, Plevrakis O, Arora S, et al. A theoretical analysis of contrastive unsupervised representation learning. In: Proceedings of the International Conference on Machine Learning. Long Beach, 2019. 5628–5637
51. Mnih A, Kavukcuoglu K. Learning word embeddings efficiently with noise-contrastive estimation. In: Proceedings of the Advances in Neural Information Processing Systems. Lake Tahoe, 2013. 2265–2273
52. Gutmann M, Hyvärinen A. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In: Proceedings of the International Conference on Artificial Intelligence and Statistics. Chia Laguna Resort, 2010. 297–304
53. Hjelm R D, Fedorov A, Lavoie-Marchildon S, et al. Learning deep representations by mutual information estimation and maximization. In: Proceedings of the International Conference on Learning Representations. New Orleans, 2019
54. Kong L, de Masson d’Autume C, Yu L, et al. A mutual information maximization perspective of language representation learning. In: Proceedings of the International Conference on Learning Representations. New Orleans, 2019
55. Clark K, Luong M T, Le Q V, et al. ELECTRA: Pre-training text encoders as discriminators rather than generators. In: Proceedings of the International Conference on Learning Representations. Addis Ababa, 2020
56. Xiong W, Du J, Wang W Y, et al. Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model. In: Proceedings of the International Conference on Learning Representations. Addis Ababa, 2020
57. Lan Z, Chen M, Goodman S, et al. ALBERT: A lite BERT for self-supervised learning of language representations. In: Proceedings of the International Conference on Learning Representations. Addis Ababa, 2020
58. de Vries W, van Cranenburgh A, Bisazza A, et al. BERTje: A Dutch BERT model. ArXiv: 1912.09582
59. Wang X, Gao T, Zhu Z, et al. KEPLER: A unified model for knowledge embedding and pre-trained language representation. ArXiv: 1911.06136
60. Radford A, Wu J, Child R, et al. Language models are unsupervised multitask learners. OpenAI Blog. 2019
61. Zhang Z, Han X, Liu Z, et al. ERNIE: enhanced language representation with informative entities. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics. Florence, 2019. 1441–1451.10.18653/v1/P19-1139 ↗
62. Peters M E, Neumann M, IV R L L, et al. Knowledge enhanced contextual word representations. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. Hong Kong, 2019. 43–5410.18653/v1/D19-1005 ↗
63. Liu W, Zhou P, Zhao Z, et al. K-BERT: Enabling language representation with knowledge graph. In: Proceedings of the AAAI Conference on Artificial Intelligence. New York, 2020. 2901–290810.1609/aaai.v34i03.5681 ↗
64. Chi Z, Dong L, Wei F, et al. Cross-lingual natural language generation via pre-training. In: Proceedings of the AAAI Conference on Artificial Intelligence. New York, 2020. 7570–757710.1609/aaai.v34i05.6256 ↗
65. Liu Y, Gu J, Goyal N, et al. Multilingual denoising pre-training for neural machine translation. ArXiv: 2001.08210
66. Conneau A, Khandelwal K, Goyal N, et al. Unsupervised cross-lingual representation learning at scale. ArXiv: 1911.02116
67. Ke P, Ji H, Liu S, et al. SentiLR: Linguistic knowledge enhanced language representation for sentiment analysis. ArXiv: 1911.02493
68. Huang H, Liang Y, Duan N, et al. Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. Hong Kong, 2019. 2485–249410.18653/v1/D19-1252 ↗
69. Eisenschlos J, Ruder S, Czapla P, et al. MultiFiT: Efficient multilingual language model fine-tuning. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. Hong Kong, 2019. 5701–570610.18653/v1/D19-1572 ↗
70. Sun Y, Wang S, Li Y, et al. ERNIE: enhanced representation through knowledge integration. ArXiv: 1904.09223
71. Cui Y, Che W, Liu T, et al. Pre-training with whole word masking for chinese BERT. ArXiv: 1906.08101
72. Wei J, Ren X, Li X, et al. NEZHA: Neural contextualized representation for chinese language understanding. ArXiv: 1909.00204
73. Diao S, Bai J, Song Y, et al. ZEN: Pre-training chinese text encoder enhanced by n-gram representations. ArXiv: 1911.00720
74. Martin L, Müller B, Suárez P J O, et al. CamemBERT: A tasty French language model. ArXiv: 1911.03894
75. Le H, Vial L, Frej J, et al. FlauBERT: Unsupervised language model pre-training for French. ArXiv: 1912.05372
76. Delobelle P, Winters T, Berendt B. RobBERT: A Dutch RoBERTa-based language model. ArXiv: 2001.06286
77. Lu J, Batra D, Parikh D, et al. ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: Proceedings of the Advances in Neural Information Processing Systems. Vancouver, 2019. 13–23
78. Tan H, Bansal M. LXMERT: Learning cross-modality encoder representations from transformers. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. Hong Kong, 2019. 5099–511010.18653/v1/D19-1514 ↗
79. Li L H, Yatskar M, Yin D, et al. VisualBERT: A simple and performant baseline for vision and language. ArXiv: 1908.03557
80. Alberti C, Ling J, Collins M, et al. Fusion of detected objects in text for visual question answering. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. Hong Kong, 2019. 2131–214010.18653/v1/D19-1219 ↗
81. Su W, Zhu X, Cao Y, et al. VL-BERT: Pre-training of generic visual-linguistic representations. In: Proceedings of the International Conference on Learning Representations. Addis Ababa, 2020
82. Sun C, Myers A, Vondrick C, et al. VideoBERT: A joint model for video and language representation learning. In: Proceedings of the International Conference on Computer Vision. Seoul, 2019. 7463–747210.1109/ICCV.2019.00756 ↗
83. Sun C, Baradel F, Murphy K, et al. Contrastive bidirectional transformer for temporal representation learning. ArXiv: 1906.05743
84. Chuang Y, Liu C, Lee H. SpeechBERT: Cross-modal pre-trained language model for end-to-end spoken question answering. ArXiv: 1910.11559
85. Lee J, Yoon W, Kim S, et al. BioBERT: A pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 2020, 36: 1234–124010.1093/bioinformatics/btz682 ↗
86. Beltagy I, Lo K, Cohan A. SciBERT: A pretrained language model for scientific text. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. Hong Kong, 2019. 3613–361810.18653/v1/D19-1371 ↗
87. Lee J and Hsiang J. PatentBERT: Patent classification with fine-tuning a pre-trained BERT model. ArXiv: 1906.02124
88. Gordon M A, Duh K, Andrews N. Compressing BERT: Studying the effects of weight pruning on transfer learning. ArXiv: 2002.08307
89. Shen S, Dong Z, Ye J, et al. Q-BERT: Hessian based ultra low precision quantization of BERT. In: Proceedings of the AAAI Conference on Artificial Intelligence. New York, 2020. 8815–882110.1609/aaai.v34i05.6409 ↗
90. Zafrir O, Boudoukh G, Izsak P, et al. Q8BERT: Quantized 8bit BERT. ArXiv: 1910.06188
91. Sanh V, Debut L, Chaumond J, et al. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. ArXiv: 1910.01108
92. Jiao X, Yin Y, Shang L, et al. TinyBERT: Distilling BERT for natural language understanding. ArXiv: 1909.10351
93. Wang W, Wei F, Dong L, et al. MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. ArXiv: 2002.10957
94. Xu C, Zhou W, Ge T, et al. BERT-of-Theseus: Compressing BERT by progressive module replacing. ArXiv: 2002.02925
95. Mikolov T, Yih W, Zweig G. Linguistic regularities in continuous space word representations. In: Proceedings of the Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics. Atlanta, 2013. 746–751
96. Rubinstein D, Levi E, Schwartz R, et al. How well do distributional models capture different types of semantic knowledge? In: Proceedings of the Annual Meeting of the Association for Computational Linguistics. Beijing, 2015. 726–73010.3115/v1/P15-2119 ↗
97. Gupta A, Boleda G, Baroni M, et al. Distributional vectors encode referential attributes. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. Lisbon, 2015. 12–2110.18653/v1/D15-1002 ↗
98. Tenney I, Xia P, Chen B, et al. What do you learn from context? Probing for sentence structure in contextualized word representations. In: Proceedings of the International Conference on Learning Representations. New Orleans, 2019
99. Liu N F, Gardner M, Belinkov Y, et al. Linguistic knowledge and transferability of contextual representations. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Minneapolis, 2019. 1073–109410.18653/v1/N19-1112 ↗
100. Tenney I, Das D, Pavlick E. BERT rediscovers the classical NLP pipeline. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics. Florence, 2019. 4593–460110.18653/v1/P19-1452 ↗