T5-Based Model for Abstractive Summarization: A Semi-Supervised Learning Approach with Consistency Loss Functions
Source abstract
Abstract
Text summarization is a prominent task in natural language processing (NLP) that condenses lengthy texts into concise summaries. Despite the success of existing supervised models, they often rely on datasets of well-constructed text pairs, which can be insufficient for languages with limited annotated data, such as Chinese. To address this issue, we propose a semi-supervised learning method for text summarization. Our method is inspired by the cycle-consistent adversarial network (CycleGAN) and considers text summarization as a style transfer task. The model is trained by using a similar procedure and loss function to those of CycleGAN and learns to transfer the style of a document to its summary and vice versa. Our method can be applied to multiple languages, but this paper focuses on its performance on Chinese documents. We trained a T5-based model and evaluated it on two datasets, CSL and LCSTS, and the results demonstrate the effectiveness of the proposed method.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Advanced Text Analysis Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Automation Science and Electrical Engineering, Beihang University, Beijing 100191, China
Science and Technology on Integrated Information System Laboratory, Institute of Software, Chinese Academy of Sciences, Beijing 100045, China
Funding
key R&D project of the Ministry of Science and Technology of the People’s Republic of China
References (38)
Show source-provided references
1. Deep reinforcement learning for extractive document summarization10.1016/j.neucom.2018.01.020 ↗
2. Sequence to sequence learning with neural networks
3. Chopra, S., Auli, M., and Rush, A.M. (2016, January 12–17). Abstractive sentence summarization with attentive recurrent neural networks. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, CA, USA.10.18653/v1/N16-1012 ↗
4. Hou, L., Hu, P., and Bei, C. (2017, January 8–12). Abstractive document summarization via neural model with joint attention. Proceedings of the National CCF Conference on Natural Language Processing and Chinese Computing, Dalian, China.10.1007/978-3-319-73618-1_28 ↗
5. Nayeem, M.T., Fuad, T.A., and Chali, Y. (2019, January 14–18). Neural diverse abstractive sentence compression generation. Proceedings of the European Conference on Information Retrieval, Cologne, Germany.10.1007/978-3-030-15719-7_14 ↗
6. Assessing sentence scoring techniques for extractive text summarization10.1016/j.eswa.2013.04.023 ↗
7. LexRank: Graph-based Lexical Centrality as Salience in Text Summarization
8. Multiple documents summarization based on evolutionary optimization algorithm10.1016/j.eswa.2012.09.014 ↗
9. Conroy, J.M., and O’Leary, D.P. (2001, January 13). Text summarization via hidden Markov models. Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, New Orleans, LA, USA.10.1145/383952.384042 ↗
10. Mihalcea, R., and Tarau, P. (2004, January 20). TextRank: Bringing Order into Texts. Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing.
11. Bollegala, D.T., Okazaki, N., and Ishizuka, M. (2005, January 11–13). A machine learning approach to sentence ordering for multidocument summarization and its evaluation. Proceedings of the International Conference on Natural Language Processing, Jeju Island, Republic of Korea.10.1007/11562214_55 ↗
12. GRAPHSUM: Discovering correlations among multiple terms for graph-based summarization10.1016/j.ins.2013.06.046 ↗
13. Cheng, J., and Lapata, M. (2016). Neural Summarization by Extracting Sentences and Words. arXiv.10.18653/v1/P16-1046 ↗
14. Nallapati, R., Zhai, F., and Zhou, B. (2016, January 12–17). SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents. Proceedings of the AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.10.1609/aaai.v31i1.10958 ↗
15. Effective Deep Learning Approaches for Summarization of Legal Texts
16. A hierarchical self-attentive neural extractive summarizer via reinforcement learning (HSASRL)10.1007/s10489-020-01669-5 ↗
17. Rush, A.M., Chopra, S., and Weston, J. (2015). A Neural Attention Model for Abstractive Sentence Summarization. arXiv.10.18653/v1/D15-1044 ↗
18. Attention Is All You Need
19. Zhang, H., Gong, Y., Yan, Y., Duan, N., Xu, J., Wang, J., Gong, M., and Zhou, M. (2019, January 21). Pretraining-Based Natural Language Generation for Text Summarization. Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), Hong Kong, China.10.18653/v1/K19-1074 ↗
20. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., and Zettlemoyer, L. (2019). BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. arXiv.10.18653/v1/2020.acl-main.703 ↗
21. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
22. Ban, H. (2004, January 4–5). Stylistic Characteristics of English News. Proceedings of the Japan-Korea Joint Symposium on Emotion & Sensibility, Daejeon, Republic of Korea.
23. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014, January 8–13). Generative Adversarial Nets. Proceedings of the Neural Information Processing Systems, Montreal, QC, Canada.
24. Zhu, J.Y., Park, T., Isola, P., and Efros, A.A. (2017, January 22–29). Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. Proceedings of the International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.244 ↗
25. Wu, R., Gu, X., Tao, X., Shen, X., Tai, Y.W., and Jia, J.I. (2019). Landmark Assisted CycleGAN for Cartoon Face Generation. arXiv.
26. Bo, C., Zhang, Q., Pan, S., and Meng, L. (2018, January 12–15). Generating Handwritten Chinese Characters using CycleGAN. Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, USA.10.1109/WACV.2018.00028 ↗
27. Gorti, S.K., and Ma, J. (2018). Text-to-Image-to-Text Translation using Cycle Consistent Adversarial Networks. arXiv.
28. Paired cycle-GAN-based image correction for quantitative cone-beam computed tomography10.1002/mp.13656 ↗
29. Kaneko, T., and Kameoka, H. (2018, January 3–7). CycleGAN-VC: Non-parallel Voice Conversion Using Cycle-Consistent Adversarial Networks. Proceedings of the 2018 26th European Signal Processing Conference (EUSIPCO), Roma, Italy.10.23919/EUSIPCO.2018.8553236 ↗
30. Kaneko, T., Kameoka, H., Tanaka, K., and Hojo, N. (2019, January 9). CycleGAN-VC2: Improved CycleGAN-based Non-parallel Voice Conversion. Proceedings of the ICASSP 2019—2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK.10.1109/ICASSP.2019.8682897 ↗
31. Kaneko, T., Kameoka, H., Tanaka, K., and Hojo, N. (2020). CycleGAN-VC3: Examining and Improving CycleGAN-VCs for Mel-spectrogram Conversion. arXiv.10.21437/Interspeech.2020-2280 ↗
32. Bishop, C. (2006). Pattern Recognition and Machine Learning, Stat Sci; Springer.
33. Li, Y., Zhang, Y., Zhao, Z., Shen, L., Liu, W., Mao, W., and Zhang, H. (2022, January 12–17). CSL: A Large-scale Chinese Scientific Literature Dataset. Proceedings of the 29th International Conference on Computational Linguistics, Gyeongju, Republic of Korea.
34. Hu, B., Chen, Q., and Zhu, F. (2015). LCSTS: A Large Scale Chinese Short Text Summarization Dataset. arXiv.10.18653/v1/D15-1229 ↗
35. CLUEbenchmark (2023, June 08). Chinese Language Generation Evaluation. Available online: https://github.com/CLUEbenchmark/CLGE.
36. Zhang, Z., Zhang, H., Chen, K., Guo, Y., Hua, J., Wang, Y., and Zhou, M. (2023, June 08). Mengzi: Towards Lightweight Yet Ingenious Pre-Trained Models for Chinese, Available online: http://xxx.lanl.gov/abs/2110.06696.
37. Lin, C.Y. (2004). Text Summarization Branches Out, Association for Computational Linguistics.
38. Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 7–12). Bleu: A Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA.10.3115/1073083.1073135 ↗