Improving Non-Autoregressive Machine Translation Using Sentence-Level Semantic Agreement
Source abstract
Abstract
Theinference stage can be accelerated significantly using a Non-Autoregressive Transformer (NAT). However, the training objective used in the NAT model also aims to minimize the loss between the generated words and the golden words in the reference. Since the dependencies between the target words are lacking, this training objective computed at word level can easily cause semantic inconsistency between the generated and source sentences. To alleviate this issue, we propose a new method, Sentence-Level Semantic Agreement (SLSA), to obtain consistency between the source and generated sentences. Specifically, we utilize contrastive learning to pull the sentence representations of the source and generated sentences closer together. In addition, to strengthen the capability of the encoder, we also integrate an agreement module into the encoder to obtain a better representation of the source sentence. The experiments are conducted on three translation datasets: the WMT 2014 EN → DE task, the WMT 2016 EN → RO task, and the IWSLT 2014 DE → DE task, and the improvement in the NAT model’s performance shows the effect of our proposed method.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Multimodal Machine Learning Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China
School of Computer Science and Technology, Beijing Institute of Technology, Beijing 100811, China
Funding
National Natural Science Foundation of China
References (39)
Show source-provided references
1. Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv.
2. Gehring, J., Auli, M., Grangier, D., Yarats, D., and Dauphin, Y.N. (2017, January 6–11). Convolutional Sequence to Sequence Learning. Proceedings of the ICML 2017, Sydney, Australia.
3. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is all you need. In Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.
4. Gu, J., Bradbury, J., Xiong, C., Li, V.O., and Socher, R. (2017). Non-autoregressive neural machine translation. arXiv.
5. Shao, C., Zhang, J., Feng, Y., Meng, F., and Zhou, J. (2020, January 7–12). Minimizing the bag-of-ngrams difference for non-autoregressive neural machine translation. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i01.5351 ↗
6. Sun, Z., Li, Z., Wang, H., Lin, Z., He, D., and Deng, Z.H. (2019). Fast structured decoding for sequence models. arXiv.
7. Qian, L., Zhou, H., Bao, Y., Wang, M., Qiu, L., Zhang, W., Yu, Y., and Li, L. (2020). Glancing transformer for non-autoregressive neural machine translation. arXiv.10.18653/v1/2021.acl-long.155 ↗
8. Saharia, C., Chan, W., Saxena, S., and Norouzi, M. (2020). Non-autoregressive machine translation with latent alignments. arXiv.10.18653/v1/2020.emnlp-main.83 ↗
9. Ghazvininejad, M., Karpukhin, V., Zettlemoyer, L., and Levy, O. (2020, January 13–18). Aligned cross entropy for non-autoregressive machine translation. Proceedings of the International Conference on Machine Learning, Virtual.
10. Gao, T., Yao, X., and Chen, D. (2021). SimCSE: Simple Contrastive Learning of Sentence Embeddings. arXiv.10.18653/v1/2021.emnlp-main.552 ↗
11. Yan, Y., Li, R., Wang, S., Zhang, F., Wu, W., and Xu, W. (2021). ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer. arXiv.10.18653/v1/2021.acl-long.393 ↗
12. Wang, R., Finch, A., Utiyama, M., and Sumita, E. (August, January 30). Sentence embedding for neural machine translation domain adaptation. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Vancouver, BC, Canada.
13. Pan, X., Wang, M., Wu, L., and Li, L. (2021). Contrastive learning for many-to-many multilingual neural machine translation. arXiv.10.18653/v1/2021.acl-long.21 ↗
14. Liu, Y., Wan, Y., Zhang, J., Zhao, W., and Yu, P. (2021, January 21–23). Enriching non-autoregressive transformer with syntactic and semantic structures for neural machine translation. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Kiev, Ukraine.10.18653/v1/2021.eacl-main.105 ↗
15. Lee, J., Mansimov, E., and Cho, K. (November, January 31). Deterministic non-autoregressive neural sequence modeling by iterative refinement. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1149 ↗
16. Sennrich, R., Haddow, B., and Birch, A. (2016, January 7–12). Neural machine translation of rare words with subword units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany.10.18653/v1/P16-1162 ↗
17. Zhou, C., Neubig, G., and Gu, J. (2019). Understanding knowledge distillation in non-autoregressive machine translation. arXiv.
18. Kim, Y., and Rush, A.M. (2016, January 1–5). Sequence-level knowledge distillation. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.10.18653/v1/D16-1139 ↗
19. A Novel Sentence-Level Agreement Architecture for Neural Machine Translation10.1109/TASLP.2020.3021347 ↗
20. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
21. Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 7–12). BLEU: A method for automatic evaluation of machine translation. Proceedings of the 40th annual meeting of the Association for Computational Linguistics, Philadelphia, PA, USA.10.3115/1073083.1073135 ↗
22. Shu, R., Lee, J., Nakayama, H., and Cho, K. (2020, January 7–12). Latent-variable non-autoregressive neural machine translation with deterministic inference using a delta posterior. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i05.6413 ↗
23. Gu, J., Wang, C., and Zhao, J. (2019). Levenshtein transformer. arXiv.
24. Ghazvininejad, M., Levy, O., Liu, Y., and Zettlemoyer, L. (2019). Mask-predict: Parallel decoding of conditional masked language models. arXiv.10.18653/v1/D19-1633 ↗
25. Guo, J., Xu, L., and Chen, E. (2020, January 5–10). Jointly masked sequence-to-sequence model for non-autoregressive neural machine translation. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Virtual.10.18653/v1/2020.acl-main.36 ↗
26. Li, Z., He, D., Tian, F., Qin, T., Wang, L., and Liu, T.Y. (2018, January 6–9). Hint-based training for non-autoregressive translation. Proceedings of the ICLR 2019 Conference, New Orleans, LA, USA.10.18653/v1/D19-1573 ↗
27. Guo, J., Tan, X., Xu, L., Qin, T., Chen, E., and Liu, T.Y. (2020, January 7–12). Fine-tuning by curriculum learning for non-autoregressive neural machine translation. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i05.6289 ↗
28. Ma, X., Zhou, C., Li, X., Neubig, G., and Hovy, E. (2019). Flowseq: Non-autoregressive conditional sequence generation with generative flow. arXiv.10.18653/v1/D19-1437 ↗
29. Bao, Y., Huang, S., Xiao, T., Wang, D., Dai, X., and Chen, J. (2021, January 6–11). Non-Autoregressive Translation by Learning Target Categorical Codes. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Virtual.10.18653/v1/2021.naacl-main.458 ↗
30. Wang, Y., Tian, F., He, D., Qin, T., Zhai, C., and Liu, T.Y. (February, January 27). Non-autoregressive machine translation with auxiliary regularization. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
31. Shao, C., Feng, Y., Zhang, J., Meng, F., Chen, X., and Zhou, J. (2019). Retrieving sequential information for non-autoregressive neural machine translation. arXiv.10.18653/v1/P19-1288 ↗
32. Huang, C., Zhou, H., Zaiane, O.R., Mou, L., and Li, L. (2021). Non-autoregressive translation with layer-wise prediction and deep supervision. arXiv.
33. Yang, M., Wang, R., Chen, K., Utiyama, M., Sumita, E., Zhang, M., and Zhao, T. (August, January 28). Sentence-level agreement for neural machine translation. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
34. Zhuang, C., Zhai, A.L., and Yamins, D. (November, January 27). Local aggregation for unsupervised learning of visual embeddings. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea.
35. Tian, Y., Krishnan, D., and Isola, P. (2020, January 23–28). Contrastive multiview coding. Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK.10.1007/978-3-030-58621-8_45 ↗
36. Hassani, K., and Khasahmadi, A.H. (2020, January 13–18). Contrastive multi-view representation learning on graphs. Proceedings of the International Conference on Machine Learning, Virtual.
37. He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. (2020, January 14–19). Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00975 ↗
38. Wu, Z., Wang, S., Gu, J., Khabsa, M., Sun, F., and Ma, H. (2020). Clear: Contrastive learning for sentence representation. arXiv.
39. Fang, H., Wang, S., Zhou, M., Ding, J., and Xie, P. (2020). Cert: Contrastive self-supervised learning for language understanding. arXiv.10.36227/techrxiv.12308378.v1 ↗