Machine Translation in Low-Resource Languages by an Adversarial Neural Network
Source abstract
Abstract
Existing Sequence-to-Sequence (Seq2Seq) Neural Machine Translation (NMT) shows strong capability with High-Resource Languages (HRLs). However, this approach poses serious challenges when processing Low-Resource Languages (LRLs), because the model expression is limited by the training scale of parallel sentence pairs. This study utilizes adversary and transfer learning techniques to mitigate the lack of sentence pairs in LRL corpora. We propose a new Low resource, Adversarial, Cross-lingual (LAC) model for NMT. In terms of the adversary technique, LAC model consists of a generator and discriminator. The generator is a Seq2Seq model that produces the translations from source to target languages, while the discriminator measures the gap between machine and human translations. In addition, we introduce transfer learning on LAC model to help capture the features in rare resources because some languages share the same subject-verb-object grammatical structure. Rather than using the entire pretrained LAC model, we separately utilize the pretrained generator and discriminator. The pretrained discriminator exhibited better performance in all experiments. Experimental results demonstrate that the LAC model achieves higher Bilingual Evaluation Understudy (BLEU) scores and has good potential to augment LRL translations.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Natural Language Processing Techniques · Topic Modeling · Text Readability and Simplification
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of ICT and Natural Sciences, Norwegian University of Science and Technology, 6009 Ålesund, Norway
Department of Computer Science, Norwegian University of Science and Technology, 2815 Gjøvik, Norway
Department of International Business, Norwegian University of Science and Technology, 6009 Ålesund, Norway
References (47)
Show source-provided references
1. Bahdanau, D., Cho, K., and Bengio, Y. (2015, January 7). Neural machine translation by jointly learning to align and translate. Proceedings of the International Conference on Learning Representations, San Diego, CA, USA.
2. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems Conference, Long Beach, CA, USA.
3. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2). Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, Minneapolis, MN, USA.
4. A survey of cross-lingual word embedding models10.1613/jair.1.11640 ↗
5. Zoph, B., Yuret, D., May, J., and Knight, K. (2016, January 1). Transfer learning for low-resource neural machine translation. Proceedings of the Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.10.18653/v1/D16-1163 ↗
6. Multi-Round Transfer Learning for Low-Resource NMT Using Multiple High-Resource Languages10.1145/3314945 ↗
7. Yong, C. (2014). Joint Training for Neural Machine Translation. [Ph.D. Thesis, IIIS Department, Tsinghua University].
8. Ren, S., Chen, W., Liu, S., Li, M., Zhou, M., and Ma, S. (2018, January 15). Triangular architecture for rare language translation. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia.10.18653/v1/P18-1006 ↗
9. Generative adversarial networks10.1145/3422622 ↗
10. Wu, L., Xia, Y., Tian, F., Zhao, L., Qin, T., Lai, J., and Liu, T.Y. (2018, January 14). Adversarial neural machine translation. Proceedings of the Asian Conference on Machine Learning (ACML 2018), Beijing, China.
11. Cao, P., Chen, Y., Liu, K., Zhao, J., and Liu, S. (2018, January 31). Adversarial transfer learning for Chinese named entity recognition with self-attention mechanism. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1017 ↗
12. Yu, L., Zhang, W., Wang, J., and Yu, Y. (2017, January 4). Seqgan: Sequence generative adversarial nets with policy gradient. Proceedings of the Thirty-first AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.10804 ↗
13. Lee, H., and Yu, T. (2018, January 15). ICASSP 2018 tutorial: Generative adversarial network; Its applications to signal processing; Natural language processing. Proceedings of the ICASSP 2018, Calgary, AB, Canada.
14. Zhang, Y., Gan, Z., Fan, K., Chen, Z., Henao, R., Shen, D., and Carin, L. (2017, January 6). Adversarial feature matching for text generation. Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia.
15. Nie, W., Narodytska, N., and Patel, A. (2018, January 30). Relgan: Relational generative adversarial networks for text generation. Proceedings of the International Conference on Learning Representations, Vancouver, BC, Canada.
16. Press, O., Bar, A., Bogin, B., Berant, J., and Wolf, L. (2017, January 6). Language generation with recurrent generative adversarial networks without pre-training. Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia.
17. Ash, J.T., and Adams, R.P. (2019). On the difficulty of warm-starting neural network training. arXiv.
18. Language-Adversarial Transfer Learning for Low-Resource Speech Recognition10.1109/TASLP.2018.2889606 ↗
19. Dai, G., Xie, J., and Fang, Y. (2017, January 23–27). Metric-based generative adversarial network. Proceedings of the 25th ACM International Conference on Multimedia (MM ‘17), Mountain View, CA, USA.10.1145/3123266.3123334 ↗
20. Dong, X., Zhu, Y., Zhang, Y., Fu, Z., Xu, D., Yang, S., and de Melo, G. (2020, January 25–30). Leveraging adversarial training in self-learning for cross-lingual text classification. Proceedings of the 43rd International ACM SIGIR Conference on Research, Development in Information Retrieval, Virtual Conference, China.10.1145/3397271.3401209 ↗
21. An investigation of credit card default prediction in the imbalanced datasets10.1109/ACCESS.2020.3033784 ↗
22. A Comparative Performance Analysis of Data Resampling Methods on Imbalance Medical Data10.1109/ACCESS.2021.3102399 ↗
23. Alam, T.M., Shaukat, K., Mahboob, H., Sarwar, M.U., Iqbal, F., Nasir, A., Hameed, I.A., and Luo, S. (2021). A Machine Learning Approach for Identification of Malignant Mesothelioma Etiological Factors in an Imbalanced Dataset. Comput. J., bxab015.10.1093/comjnl/bxab015 ↗
24. Latif, M.Z., Shaukat, K., Luo, S., Hameed, I.A., Iqbal, F., and Alam, T.M. (2020, January 12–13). Risk factors identification of malignant mesothelioma: A data mining based approach. Proceedings of the 2020 International Conference on Electrical, Communication, and Computer Engineering (ICECCE), Istanbul, Turkey.10.1109/ICECCE49384.2020.9179443 ↗
25. Yang, X., Khushi, M., and Shaukat, K. (2020, January 16–18). Biomarker CA125 Feature engineering and class imbalance learning improves ovarian cancer prediction. Proceedings of the 2020 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE), Gold Coast, Australia.10.1109/CSDE50874.2020.9411607 ↗
26. Li, Y., Jiang, J., Yangji, J., and Ma, N. (2021, January 16–18). Finding better subwords for Tibetan neural machine translation. Proceedings of the Transactions on Asian and Low-Resource Language Information Processing, Gold Coast, Australia.10.1145/3448216 ↗
27. Word re-segmentation in Chinese-Vietnamese machine translation10.1145/2988237 ↗
28. Choi, Y.-S., Park, Y.-H., Yun, S., Kim, S.-H., and Lee, K.-J. (2021). Factors Behind the Effectiveness of an Unsupervised Neural Machine Translation System between Korean and Japanese. Appl. Sci., 11.10.3390/app11167662 ↗
29. Nguyen, T.Q., and Chiang, D. (2019, January 3). Zero-shot reading comprehension by cross-lingual transfer learning with multi-lingual language representation model. Proceedings of the EMNLP, Hong Kong, China.
30. Arjovsky, M., Chintala, S., and Bottou, L. (2017, January 6). Wasserstein generative adversarial networks. Proceedings of the International Conference on Machine Learning, Sydney, Australia.
31. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. (2017, January 4). Improved training of Wasserstein GANs. Proceedings of the Advances in Neural Information Processing Systems Conference, Long Beach, CA, USA.
32. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., and Dean, J. (2013, January 5). Distributed representations of words and phrases and their compositionality. Proceedings of the Advances in Neural Information Processing Systems Conference, Lake Tahoe, NV, USA.
33. Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014, January 25–29). Learning phrase representations using RNN encoder–decoder for statistical machine translation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP 2014), Doha, Qatar.10.3115/v1/D14-1179 ↗
34. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR 2016), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
35. Clevert, D.A., Unterthiner, T., and Hochreiter, S. (2016, January 2). Fast and accurate deep network learning by Exponential Linear Units (ELUs). Proceedings of the International Conference on Learning Representations (ICLR 2016), San Juan, Puerto Rico.
36. Kim, J., Won, M., Serra, X., and Liem, C.C. (2018, January 23). Transfer learning of artist group factors to musical genre classification. Proceedings of the Web Conference 2018 (WWW ‘18), Lyon, France.10.1145/3184558.3191823 ↗
37. Veit, A., Wilber, M.J., and Belongie, S. (2016, January 5). Residual networks behave like ensembles of relatively shallow networks. Proceedings of the Advances in Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.
38. Balduzzi, D., Frean, M., Leary, L., Lewis, J.P., Ma, K.W.D., and McWilliams, B. (2017, January 6). The shattered gradients problem: If resnets are the answer, then what is the question?. Proceedings of the 34th International Conference on Machine Learning (ICML’17), Sydney, Australia.
39. Fu, Z., Xian, Y., Geng, S., Ge, Y., Wang, Y., Dong, X., Wang, G., and de Melo, G. (2017, January 7). ABSent: Cross-lingual sentence representation mapping with bidirectional GANs. Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI 2020), New York, NY, USA.10.1609/aaai.v34i05.6279 ↗
40. Tran, C., Tang, Y., Li, X., and Gu, J. (2020, January 6). Cross-lingual retrieval for iterative self-supervised training. Proceedings of the Advances in Neural Information Processing Systems Conference (NIPS 2020), Virtual Conference.
41. Mathur, N., Baldwin, T., and Cohn, T. (2020, January 5). Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020), Virtual Conference.10.18653/v1/2020.acl-main.448 ↗
42. Popović, M. (2015, January 17). chrF: Character n-gram F-score for automatic MT evaluation. Proceedings of the Tenth Workshop on Statistical Machine Translation (EMNLP 2015 Workshop), Lisbon, Portugal.10.18653/v1/W15-3049 ↗
43. A Survey on Machine Learning Techniques for Cyber Security in the Last Decade10.1109/ACCESS.2020.3041951 ↗
44. Shaukat, K., Luo, S., Varadharajan, V., Hameed, I.A., Chen, S., Liu, D., and Li, J. (2020). Performance Comparison and Current Challenges of Using Machine Learning Techniques in Cybersecurity. Energies, 13.10.3390/en13102509 ↗
45. Jean, S., Cho, K., Memisevic, R., and Bengio, Y. (2015, January 26). On using very large target vocabulary for neural machine translation. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2015), Beijing, China.10.3115/v1/P15-1001 ↗
46. Zhu, J., Xia, Y., Wu, L., He, D., Qin, T., Zhou, W., Li, H., and Liu, T.Y. (2020, January 30). Incorporating BERT into neural machine translation. Proceedings of the International Conference on Learning Representations (ICLR 2020), Virtual Conference.
47. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (2020, January 30). ALBERT: A lite BERT for self-supervised learning of language representations. Proceedings of the International Conference on Learning Representations (ICLR 2020), Virtual Conference.