Boosting the Transformer with the BERT Supervision in Low-Resource Machine Translation
Source abstract
Abstract
Previous works trained the Transformer and its variants end-to-end and achieved remarkable translation performance when there are huge parallel sentences available. However, these models suffer from the data scarcity problem in low-resource machine translation tasks. To deal with the mismatch problem between the big model capacity of the Transformer and the small parallel training data set, this paper adds the BERT supervision on the latent representation between the encoder and the decoder of the Transformer and designs a multi-step training algorithm to boost the Transformer on such a basis. The algorithm includes three stages: (1) encoder training, (2) decoder training, and (3) joint optimization. We introduce the BERT of the target language in the encoder and the decoder training and alleviate the data starvation problem of the Transformer. After the training stage, the BERT will not further attend the inference section explicitly. Another merit of our training algorithm is that it can further enhance the Transformer in the task where there are limited parallel sentence pairs but large amounts of monolingual corpus of the target language. The evaluation results on six low-resource translation tasks suggest that the Transformer trained by our algorithm significantly outperforms the baselines which were trained end-to-end in previous works.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Natural Language Processing Techniques · Topic Modeling · Multimodal Machine Learning Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Computer Science, Inner Mongolia University, Hohhot 010020, China
Funding
National Natural Science Foundation of China · Key Technology Research Program of Inner Mongolia Autonomous Region · Key R&D and Achievement Transformation Program of Inner Mongolia Autonomous Region · Big Data Lab of Inner Mongolia Discipline Inspection and Supervision Committee
References (46)
Show source-provided references
1. Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv.
2. Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014, January 25–29). Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1179 ↗
3. Attention is all you need
4. Luong, T., Pham, H., and Manning, C.D. (2015, January 17–21). Effective Approaches to Attention-based Neural Machine Translation. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal.10.18653/v1/D15-1166 ↗
5. Sun, H., Wang, R., Chen, K., Utiyama, M., Sumita, E., and Zhao, T. (August, January 28). Unsupervised Bilingual Word Embedding Agreement for Unsupervised Neural Machine Translation. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
6. Britz, D., Goldie, A., Luong, M.T., and Le, Q. (2017, January 7–11). Massive Exploration of Neural Machine Translation Architectures. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.10.18653/v1/D17-1151 ↗
7. Gehring, J., Auli, M., Grangier, D., Yarats, D., and Dauphin, Y.N. (2017, January 6–11). Convolutional sequence to sequence learning. Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia.
8. Ramesh, S.H., and Sankaranarayanan, K.P. (2018, January 2–4). Neural Machine Translation for Low Resource Languages using Bilingual Lexicon Induced from Comparable Corpora. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, New Orleans, LA, USA.10.18653/v1/N18-4016 ↗
9. Lignos, C., Cohen, D., Lien, Y.C., Mehta, P., Croft, W.B., and Miller, S. (2019, January 3–7). The Challenges of Optimizing Machine Translation for Low Resource Cross-Language Information Retrieval. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.10.18653/v1/D19-1353 ↗
10. Nguyen, T.Q., and Chiang, D. (December, January 27). Transfer Learning across Low-Resource, Related Languages for Neural Machine Translation. Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), Taipei, Taiwan.
11. Kim, Y., Gao, Y., and Ney, H. (August, January 28). Effective Cross-lingual Transfer of Neural Machine Translation Models without Shared Vocabularies. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
12. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2–7). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, MN, USA.
13. Pan, X., Wang, M., Wu, L., and Li, L. (2021, January 1–6). Contrastive Learning for Many-to-many Multilingual Neural Machine Translation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Virtual Event.10.18653/v1/2021.acl-long.21 ↗
14. Sennrich, R., Haddow, B., and Birch, A. (2016, January 7–12). Improving Neural Machine Translation Models with Monolingual Data. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany.10.18653/v1/P16-1009 ↗
15. The capacity of feedforward neural networks10.1016/j.neunet.2019.04.009 ↗
16. Zhu, J., Xia, Y., Wu, L., He, D., Qin, T., Zhou, W., Li, H., and Liu, T. (2019, January 6–9). Incorporating BERT into Neural Machine Translation. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA.
17. Song, K., Tan, X., Qin, T., Lu, J., and Liu, T.Y. (2019, January 10–15). MASS: Masked Sequence to Sequence Pre-training for Language Generation. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA.
18. Cross-lingual language model pretraining
19. Clinchant, S., Jung, K.W., and Nikoulina, V. (2019, January 4). On the use of BERT for Neural Machine Translation. Proceedings of the 3rd Workshop on Neural Generation and Translation, Hong Kong, China.10.18653/v1/D19-5611 ↗
20. Sequence to sequence learning with neural networks
21. Weng, R., Wei, H., Huang, S., Yu, H., Bing, L., Luo, W., and Chen, J. (2020, January 7–12). Gret: Global representation enhanced transformer. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i05.6464 ↗
22. Bapna, A., Chen, M., Firat, O., Cao, Y., and Wu, Y. (November, January 31). Training Deeper Neural Machine Translation Models with Transparent Attention. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1338 ↗
23. Wang, Q., Li, B., Xiao, T., Zhu, J., Li, C., Wong, D.F., and Chao, L.S. (August, January 28). Learning Deep Transformer Models for Machine Translation. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
24. Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018, January 1–6). Deep contextualized word representations. Proceedings of the NAACL-HLT, New Orleans, LA, USA.10.18653/v1/N18-1202 ↗
25. Xlnet: Generalized autoregressive pretraining for language understanding
26. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv.
27. Language models are unsupervised multitask learners
28. Yang, J., Wang, M., Zhou, H., Zhao, C., Zhang, W., Yu, Y., and Li, L. (2020, January 7–12). Towards making the most of bert in neural machine translation. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.
29. Zoph, B., Yuret, D., May, J., and Knight, K. (2016, January 1–5). Transfer Learning for Low-Resource Neural Machine Translation. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.10.18653/v1/D16-1163 ↗
30. Ahmadnia, B., Serrano, J., and Haffari, G. (2017, January 2–8). Persian-Spanish Low-Resource Statistical Machine Translation Through English as Pivot Language. Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP, Varna, Bulgaria.10.26615/978-954-452-049-6_004 ↗
31. He, D., Xia, Y., Qin, T., Wang, L., Yu, N., Liu, T.Y., and Ma, W.Y. (2016, January 5–10). Dual learning for machine translation. Proceedings of the 30th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.
32. Augmenting neural machine translation through round-trip training approach10.1515/comp-2019-0019 ↗
33. Spanish-Turkish Low-Resource Machine Translation: Unsupervised Learning vs Round-Tripping
34. Chronopoulou, A., Stojanovski, D., and Fraser, A. (2021). Improving the lexical ability of pretrained language models for unsupervised neural machine translation. arXiv.10.18653/v1/2021.naacl-main.16 ↗
35. Atrio, À.R., and Popescu-Belis, A. (2022, January 1–3). On the Interaction of Regularization Factors in Low-resource Neural Machine Translation. Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, Ghent, Belgium.
36. Qi, Y., Sachan, D., Felix, M., Padmanabhan, S., and Neubig, G. (2018, January 1–6). When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation?. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), New Orleans, LA, USA.10.18653/v1/N18-2084 ↗
37. Wang, Y., Zhai, C., and Awadalla, H.H. (2020, January 16–20). Multi-task Learning for Multilingual Neural Machine Translation. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.10.18653/v1/2020.emnlp-main.75 ↗
38. Tang, Y., Tran, C., Li, X., Chen, P.J., Goyal, N., Chaudhary, V., Gu, J., and Fan, A. (2021, January 1–6). Multilingual translation from denoising pre-training. Proceedings of the Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, Online.10.18653/v1/2021.findings-acl.304 ↗
39. Chi, Z., Dong, L., Ma, S., Huang, S., Singhal, S., Mao, X.L., Huang, H.Y., Song, X., and Wei, F. (2021, January 7–11). mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online.10.18653/v1/2021.emnlp-main.125 ↗
40. Leveraging pre-trained checkpoints for sequence generation tasks10.1162/tacl_a_00313 ↗
41. Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 6–12). BLEU: A method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA.10.3115/1073083.1073135 ↗
42. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., and Funtowicz, M. (2019). HuggingFace’s Transformers: State-of-the-art Natural Language Processing. arXiv.10.18653/v1/2020.emnlp-demos.6 ↗
43. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
44. Sennrich, R., Haddow, B., and Birch, A. (2015). Neural Machine Translation of Rare Words with Subword Units. Comput. Sci.10.18653/v1/P16-1162 ↗
45. Dou, Z.Y., Tu, Z., Wang, X., Shi, S., and Zhang, T. (November, January 31). Exploiting Deep Representations for Neural Machine Translation. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1457 ↗
46. Shaw, P., Uszkoreit, J., and Vaswani, A. (2018, January 1–6). Self-Attention with Relative Position Representations. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), New Orleans, LA, USA.10.18653/v1/N18-2074 ↗