Reusing Monolingual Pre-Trained Models by Cross-Connecting Seq2seq Models for Machine Translation
Source abstract
Abstract
This work uses sequence-to-sequence (seq2seq) models pre-trained on monolingual corpora for machine translation. We pre-train two seq2seq models with monolingual corpora for the source and target languages, then combine the encoder of the source language model and the decoder of the target language model, i.e., the cross-connection. We add an intermediate layer between the pre-trained encoder and the decoder to help the mapping of each other since the modules are pre-trained completely independently. These monolingual pre-trained models can work as a multilingual pre-trained model because one model can be cross-connected with another model pre-trained on any other language, while their capacity is not affected by the number of languages. We will demonstrate that our method improves the translation performance significantly over the random baseline. Moreover, we will analyze the appropriate choice of the intermediate layer, the importance of each part of a pre-trained model, and the performance change along with the size of the bitext.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Natural Language Processing Techniques · Topic Modeling · Text Readability and Simplification
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Artificial Intelligence Laboratory, Hanyang University, Seoul 04763, Korea
Funding
National Research Foundation of Korea · Ministry of Science and ICT
References (51)
Show source-provided references
1. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
2. Multilingual denoising pre-training for neural machine translation10.1162/tacl_a_00343 ↗
3. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. (2019). Unsupervised cross-lingual representation learning at scale. arXiv.10.18653/v1/2020.acl-main.747 ↗
4. Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. (2020). mT5: A massively multilingual pre-trained text-to-text transformer. arXiv.10.18653/v1/2021.naacl-main.41 ↗
5. Carmo, D., Piau, M., Campiotti, I., Nogueira, R., and Lotufo, R. (2020). PTT5: Pretraining and validating the T5 model on Brazilian Portuguese data. arXiv.
6. de Vries, W., van Cranenburgh, A., Bisazza, A., Caselli, T., van Noord, G., and Nissim, M. (2019). Bertje: A dutch bert model. arXiv.
7. Delobelle, P., Winters, T., and Berendt, B. (2020). Robbert: A dutch roberta-based language model. arXiv.10.18653/v1/2020.findings-emnlp.292 ↗
8. Dumitrescu, S.D., Avram, A.M., and Pyysalo, S. (2020). The birth of Romanian BERT. arXiv.10.18653/v1/2020.findings-emnlp.387 ↗
9. Kuratov, Y., and Arkhipov, M. (2019). Adaptation of deep bidirectional multilingual transformers for russian language. arXiv.
10. Martin, L., Muller, B., Suárez, P.J.O., Dupont, Y., Romary, L., de La Clergerie, É.V., Seddah, D., and Sagot, B. (2019). Camembert: A tasty french language model. arXiv.10.18653/v1/2020.acl-main.645 ↗
11. Le, H., Vial, L., Frej, J., Segonne, V., Coavoux, M., Lecouteux, B., Allauzen, A., Crabbé, B., Besacier, L., and Schwab, D. (2019). Flaubert: Unsupervised language model pre-training for french. arXiv.
12. Louis, A. (2021, September 17). BelGPT-2: A GPT-2 Model Pre-Trained on French Corpora. Available online: https://github.com/antoiloui/belgpt2.
13. Polignano, M., Basile, P., De Gemmis, M., Semeraro, G., and Basile, V. (2019, January 13–15). Alberto: Italian BERT language understanding model for NLP challenging tasks based on tweets. Proceedings of the 6th Italian Conference on Computational Linguistics, CLiC-it 2019, Bari, Italy.
14. De Mattei, L., Cafagna, M., Dell’Orletta, F., Nissim, M., and Guerini, M. (2020). Geppetto carves italian into a language model. arXiv.10.4000/books.aaccademia.8438 ↗
15. Souza, F., Nogueira, R., and Lotufo, R. (2019). Portuguese named entity recognition using BERT-CRF. arXiv.
16. Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv.10.3115/v1/D14-1179 ↗
17. Lample, G., and Conneau, A. (2019). Cross-lingual language model pretraining. arXiv.
18. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. (2021, September 17). Improving Language Understanding by Generative Pre-Training; 2018. Available online: https://www.cs.ubc.ca/~amuham01/LING530/papers/radford2018improving.pdf.
19. Tang, Y., Tran, C., Li, X., Chen, P.J., Goyal, N., Chaudhary, V., Gu, J., and Fan, A. (2020). Multilingual translation with extensible multilingual pretraining and finetuning. arXiv.
20. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. arXiv.
21. Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep contextualized word representations. arXiv.10.18653/v1/N18-1202 ↗
22. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv.
23. Spanbert: Improving pre-training by representing and predicting spans10.1162/tacl_a_00300 ↗
24. Song, K., Tan, X., Qin, T., Lu, J., and Liu, T.Y. (2019). Mass: Masked sequence to sequence pre-training for language generation. arXiv.
25. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2019). Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv.10.18653/v1/2020.acl-main.703 ↗
26. Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., and Hon, H.W. (2019). Unified language model pre-training for natural language understanding and generation. arXiv.
27. Bao, H., Dong, L., Wei, F., Wang, W., Yang, N., Liu, X., Wang, Y., Gao, J., Piao, S., and Zhou, M. (2020, January 13–18). Unilmv2: Pseudo-masked language models for unified language model pre-training. Proceedings of the International Conference on Machine Learning, PMLR 2020, Virtual.
28. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P.J. (2019). Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv.
29. Baziotis, C., Titov, I., Birch, A., and Haddow, B. (2021). Exploring Unsupervised Pretraining Objectives for Machine Translation. Findings of the Association for Computational Linguistics, Proceedings of the ACL-IJCNLP 2021, Online, 1–6 August 2021, Association for Computational Linguistics.10.18653/v1/2021.findings-acl.261 ↗
30. Leveraging pre-trained checkpoints for sequence generation tasks10.1162/tacl_a_00313 ↗
31. Zhu, J., Xia, Y., Wu, L., He, D., Qin, T., Zhou, W., Li, H., and Liu, T.Y. (2020). Incorporating bert into neural machine translation. arXiv.
32. Yang, J., Wang, M., Zhou, H., Zhao, C., Zhang, W., Yu, Y., and Li, L. (2020, January 7–12). Towards making the most of bert in neural machine translation. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.
33. Imamura, K., and Sumita, E. (2019, January 4). Recycling a pre-trained BERT encoder for neural machine translation. Proceedings of the 3rd Workshop on Neural Generation and Translation, Hong Kong, China.10.18653/v1/D19-5603 ↗
34. Clinchant, S., Jung, K.W., and Nikoulina, V. (2019). On the use of bert for neural machine translation. arXiv.10.18653/v1/D19-5611 ↗
35. Vázquez, R., Celikkanat, H., Creutz, M., and Tiedemann, J. (2021, January 5–6). On the differences between BERT and MT encoder spaces and how to address them in translation tasks. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop, Online.10.18653/v1/2021.acl-srw.35 ↗
36. Yoo, J., Eom, H., and Choi, Y.S. (2019). Image-To-Image Translation Using a Cross-Domain Auto-Encoder and Decoder. Appl. Sci., 9.10.3390/app9224780 ↗
37. Firat, O., Cho, K., and Bengio, Y. (2016). Multi-way, multilingual neural machine translation with a shared attention mechanism. arXiv.10.18653/v1/N16-1101 ↗
38. Aharoni, R., Johnson, M., and Firat, O. (2019). Massively multilingual neural machine translation. arXiv.10.18653/v1/N19-1388 ↗
39. Lu, Y., Keung, P., Ladhak, F., Bhardwaj, V., Zhang, S., and Sun, J. (2018). A neural interlingua for multilingual machine translation. arXiv.10.18653/v1/W18-6309 ↗
40. Lewis, M., Ghazvininejad, M., Ghosh, G., Aghajanyan, A., Wang, S., and Zettlemoyer, L. (2020). Pre-training via paraphrasing. arXiv.
41. Chi, Z., Dong, L., Ma, S., Mao, S.H.X.L., Huang, H., and Wei, F. (2021). mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs. arXiv.10.18653/v1/2021.emnlp-main.125 ↗
42. Artetxe, M., Ruder, S., and Yogatama, D. (2019). On the cross-lingual transferability of monolingual representations. arXiv.10.18653/v1/2020.acl-main.421 ↗
43. Yu, H., Chen, C., Du, X., Li, Y., Rashwan, A., Hou, L., Jin, P., Yang, F., Liu, F., and Kim, J. (2021, September 17). TensorFlow Model Garden. Available online: https://github.com/tensorflow/models.
44. Bojar, O.r., Chatterjee, R., Federmann, C., Graham, Y., Haddow, B., Huang, S., Huck, M., Koehn, P., Liu, Q., and Logacheva, V. (2017, January 7–11). Findings of the 2017 Conference on Machine Translation (WMT17). Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, Copenhagen, Denmark.10.18653/v1/W17-4717 ↗
45. (2021, September 17). ParaCrawl. Available online: https://paracrawl.eu/.
46. Bojar, O.r., Chatterjee, R., Federmann, C., Graham, Y., Haddow, B., Huck, M., Jimeno Yepes, A., Koehn, P., Logacheva, V., and Monz, C. (2016, January 11–12). Findings of the 2016 Conference on Machine Translation. Proceedings of the First Conference on Machine Translation, Berlin, Germany.10.18653/v1/W16-2301 ↗
47. Cettolo, M., Girardi, C., and Federico, M. (2012, January 28–30). Wit3: Web inventory of transcribed and translated talks. Proceedings of the 16th Conference of European Association for Machine Translation, Trento, Italy.
48. Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., and Auli, M. (2019, January 2–7). fairseq: A Fast, Extensible Toolkit for Sequence Modeling. Proceedings of the NAACL-HLT 2019: Demonstrations, Minneapolis, MN, USA.10.18653/v1/N19-4009 ↗
49. Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., and Macherey, K. (2016). Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv.
50. Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 7–12). Bleu: A method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadephia, PA, USA.10.3115/1073083.1073135 ↗
51. Dalvi, F., Durrani, N., Sajjad, H., Belinkov, Y., and Vogel, S. (December, January 27). Understanding and Improving Morphological Learning in the Neural Machine Translation Decoder. Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Taipei, Taiwan.