A Multitask Cross-Lingual Summary Method Based on ABO Mechanism
Source abstract
Abstract
Recent cross-lingual summarization research has pursued the use of a unified end-to-end model which has demonstrated a certain level of improvement in performance and effectiveness, but this approach stitches together multiple tasks and makes the computation more complex. Less work has focused on alignment relationships across languages, which has led to persistent problems of summary misordering and loss of key information. For this reason, we first simplify the multitasking by converting the translation task into an equal proportion of cross-lingual summary tasks so that the model can perform only cross-lingual summary tasks when generating cross-lingual summaries. In addition, we splice monolingual and cross-lingual summary sequences as an input so that the model can fully learn the core content of the corpus. Then, we propose a reinforced regularization method based on the model to improve its robustness, and build a targeted ABO mechanism to enhance the semantic relationship alignment and key information retention of the cross-lingual summaries. Ablation experiments are conducted on three datasets of different orders of magnitude to demonstrate the effective enhancement of the model by the optimization approach; they outperform the mainstream approaches on the cross-lingual summarization task and the monolingual summarization task for the full dataset. Finally, we validate the model’s capabilities on a cross-lingual summary dataset of professional domains, and the results demonstrate its superior performance and ability to improve cross-lingual sequencing.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Data Quality and Management
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Electronic and Electrical Engineering, Shanghai University of Engineering Science, Shanghai 201620, China
Department of Automation, Shanghai Jiao Tong University, Shanghai 200240, China
Funding
Science and Technology Innovation 2030—Major Project of “New Generation Artificial Intelligence” · 2022 Major R&D Special 03 and 5G Projects of Jiangxi Provincial Department of Science and Technology of China
References (35)
Show source-provided references
1. A survey on cross-lingual summarization10.1162/tacl_a_00520 ↗
2. Mohammadzadeh, A., Sabzalian, M.H., Zhang, C., Castillo, O., Sakthivel, R., and El-Sousy, F.F. (2022). Modern Adaptive Fuzzy Control Systems, Springer Nature.10.1007/978-3-031-17393-6 ↗
3. Wan, X. (2011, January 19–24). Using bilingual information for cross-language document summarization. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, OR, USA.
4. Abstractive cross-language summarization via translation model enhanced predicate argument structure fusing10.1109/TASLP.2016.2586608 ↗
5. Pires, T., Schlinger, E., and Garrette, D. (2019). How multilingual is multilingual BERT?. arXiv.10.18653/v1/P19-1493 ↗
6. Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. (2020). mT5: A massively multilingual pre-trained text-to-text transformer. arXiv.10.18653/v1/2021.naacl-main.41 ↗
7. Mohammadazadeh, A., Sabzalian, M.H., Castillo, O., Sakthivel, R., El-Sousy, F.F., and Mobayen, S. (2022). Neural Networks and Learning Algorithms in MATLAB, Springer Nature.10.1007/978-3-031-14571-1 ↗
8. Joachims, T. (2002, January 23–26). Optimizing search engines using clickthrough data. Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Edmonton, AB, Canada.10.1145/775047.775067 ↗
9. See, A., Liu, P.J., and Manning, C.D. (2017). Get to the point: Summarization with pointer-generator networks. arXiv.10.18653/v1/P17-1099 ↗
10. Cross-lingual c* st* rd: English access to hindi information10.1145/979872.979877 ↗
11. Ouyang, J., Song, B., and McKeown, K. (2019, January 1). A robust abstractive system for cross-lingual summarization. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, MN, USA.10.18653/v1/N19-1204 ↗
12. Cross-language document summarization via extraction and ranking of multiple summaries10.1007/s10115-018-1152-7 ↗
13. Lim, J.M., Kang, I.S., and Lee, J.H. (2004, January 2–4). Multi-Document Summarization Using Cross-Language Texts. Proceedings of the NTCIR, Tokyo, Japan.
14. Orašan, C., and Chiorean, O.A. (June, January 26). Evaluation of a cross-lingual romanian-english multi-document summariser. Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco.
15. Wan, X., Li, H., and Xiao, J. (2010, January 11–16). Cross-language document summarization based on machine translation quality prediction. Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, Uppsala, Sweden.
16. Cao, Y., Liu, H., and Wan, X. (2020, January 5–10). Jointly learning to align and summarize for neural cross-lingual summarization. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.554 ↗
17. Zhu, J., Zhou, Y., Zhang, J., and Zong, C. (2020, January 5–10). Attend, translate and summarize: An efficient method for neural cross-lingual summarization. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.121 ↗
18. Luo, F., Wang, W., Liu, J., Liu, Y., Bi, B., Huang, S., Huang, F., and Si, L. (2020). VECO: Variable and flexible cross-lingual pre-training for language understanding and generation. arXiv.10.18653/v1/2021.acl-long.308 ↗
19. Zhu, J., Wang, Q., Wang, Y., Zhou, Y., Zhang, J., Wang, S., and Zong, C. (2019). NCLS: Neural cross-lingual summarization. arXiv.10.18653/v1/D19-1302 ↗
20. Bai, Y., Gao, Y., and Huang, H. (2021). Cross-lingual abstractive summarization with limited parallel resources. arXiv.10.18653/v1/2021.acl-long.538 ↗
21. Liang, Y., Meng, F., Zhou, C., Xu, J., Chen, Y., Su, J., and Zhou, J. (2022). A variational hierarchical model for neural cross-lingual summarization. arXiv.10.18653/v1/2022.acl-long.148 ↗
22. Sohn, K., Lee, H., and Yan, X. (2015, January 7–12). Learning structured output representation using deep conditional generative models. Proceedings of the 28th International Conference on Neural Information Processing Systems, Montreal, QC, Canada.
23. Duan, X., Yu, H., Yin, M., Zhang, M., Luo, W., and Zhang, Y. (2019). Contrastive attention mechanism for abstractive sentence summarization. arXiv.10.18653/v1/D19-1301 ↗
24. Jiang, S., Tu, D., Chen, X., Tang, R., Wang, W., and Wang, H. (2022). CptGraphSum: Let key clues guide the cross-lingual abstractive summarization. arXiv.
25. On information and sufficiency10.1214/aoms/1177729694 ↗
26. Bishop, C.M., and Nasrabadi, N.M. (2006). Pattern Recognition and Machine Learning, Springer.
27. Chen, Z., Badrinarayanan, V., Lee, C.Y., and Rabinovich, A. (2018, January 10–15). Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. Proceedings of the International Conference on Machine Learning, PMLR, Singapore.
28. Hermann, K.M., Kocisky, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., and Blunsom, P. (2015, January 7–12). Teaching machines to read and comprehend. Proceedings of the 28th International Conference on Neural Information Processing Systems, Montreal, QC, Canada.
29. Zhu, J., Li, H., Liu, T., Zhou, Y., Zhang, J., and Zong, C. (November, January 31). MSMO: Multimodal summarization with multimodal output. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1448 ↗
30. Hu, B., Chen, Q., and Zhu, F. (2015). Lcsts: A large scale chinese short text summarization dataset. arXiv.10.18653/v1/D15-1229 ↗
31. Abdaoui, A., Pradel, C., and Sigel, G. (2020). Load what you need: Smaller versions of multilingual bert. arXiv.10.18653/v1/2020.sustainlp-1.16 ↗
32. Shazeer, N., and Stern, M. (July, January 25). Adafactor: Adaptive learning rates with sublinear memory cost. Proceedings of the International Conference on Machine Learning, PMLR, Vienna, Austria.
33. Lin, C.Y. (2004). Text Summarization Branches Out, Association for Computational Linguistics.
34. Framework for deep learning-based language models using multi-task learning in natural language understanding: A systematic literature review and future directions10.1109/ACCESS.2022.3149798 ↗
35. Alfassy, A., Arbelle, A., Halimi, O., Harary, S., Herzig, R., Schwartz, E., Panda, R., Dolfi, M., Auer, C., and Saenko, K. (2022). FETA: Towards Specializing Foundation Models for Expert Task Applications. arXiv.