Attentional Extractive Summarization
Source abstract
Abstract
In this work, a general theoretical framework for extractive summarization is proposed—the Attentional Extractive Summarization framework. Although abstractive approaches are generally used in text summarization today, extractive methods can be especially suitable for some applications, and they can help with other tasks such as Text Classification, Question Answering, and Information Extraction. The proposed approach is based on the interpretation of the attention mechanisms of hierarchical neural networks, which compute document-level representations of documents and summaries from sentence-level representations, which, in turn, are computed from word-level representations. The models proposed under this framework are able to automatically learn relationships among document and summary sentences, without requiring Oracle systems to compute the reference labels for each sentence before the training phase. These relationships are obtained as a result of a binary classification process, the goal of which is to distinguish correct summaries for documents. Two different systems, formalized under the proposed framework, were evaluated on the CNN/DailyMail and the NewsRoom corpora, which are some of the reference corpora in the most relevant works on text summarization. The results obtained during the evaluation support the adequacy of our proposal and suggest that there is still room for the improvement of our attentional framework.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Advanced Text Analysis Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Valencian Research Institute for Artificial Intelligence (VRAIN), Universitat Politècnica de València, 46022 Valencia, Spain
Funding
MCIN/AEI/10.13039/501100011033 · European Union “NextGenerationEU/PRTR”
References (45)
Show source-provided references
1. LexRank: Graph-based lexical centrality as salience in text summarization10.1613/jair.1523 ↗
2. Mihalcea, R., and Tarau, P. (2004, January 25–26). TextRank: Bringing Order into Text. Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, Barcelona, Spain.
3. Extractive summarization using complex networks and syntactic dependency10.1016/j.physa.2011.10.015 ↗
4. Ferreira, R., Freitas, F., de Souza Cabral, L., Lins, R.D., Lima, R., França, G., Simskez, S.J., and Favaro, L. (2013, January 17–20). A four dimension graph model for automatic text summarization. Proceedings of the IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT) IEEE, Atlanta, GA, USA.10.1109/WI-IAT.2013.55 ↗
5. Extractive multi-document summarization using multilayer networks10.1016/j.physa.2018.03.013 ↗
6. Neural summarization by extracting sentences and words10.18653/v1/P16-1046 ↗
7. Nallapati, R., Zhai, F., and Zhou, B. (2017, January 4–9). Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI’17, San Francisco, CA, USA.10.1609/aaai.v31i1.10958 ↗
8. Liu, Y., and Lapata, M. (2019, January 3–7). Text summarization with pretrained encoders. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.10.18653/v1/D19-1387 ↗
9. Zhong, M., Liu, P., Chen, Y., Wang, D., Qiu, X., and Huang, X. (2020, January 5–10). Extractive summarization as text matching. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.552 ↗
10. Ranking sentences for extractive summarization with reinforcement learning
11. Zhang, X., Lapata, M., Wei, F., and Zhou, M. (November, January 31). Neural latent extractive document summarization. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1088 ↗
12. Dong, Y., Shen, Y., Crawford, E., van Hoof, H., and Cheung, J.C.K. (November, January 31). BanditSum: Extractive summarization as a contextual bandit. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1409 ↗
13. Deep reinforcement learning for extractive document summarization10.1016/j.neucom.2018.01.020 ↗
14. Fast abstractive summarization with reinforce-selected sentence rewriting10.18653/v1/P18-1063 ↗
15. Summcoder: An unsupervised framework for extractive text summarization based on deep auto-encoders10.1016/j.eswa.2019.03.045 ↗
16. Text document summarization using word embedding10.1016/j.eswa.2019.112958 ↗
17. Lin, C.Y. (2004). Text Summarization Branches Out, Association for Computational Linguistics.
18. Siamese hierarchical attention networks for extractive summarization10.3233/JIFS-179011 ↗
19. Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., and Hovy, E. (2016, January 12–17). Hierarchical attention networks for document classification. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, CA, USA.10.18653/v1/N16-1174 ↗
20. Extractive summarization using Siamese hierarchical transformer encoders10.3233/JIFS-179901 ↗
21. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, Long Beach, CA, USA.
22. Teaching machines to read and comprehend
23. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies
24. González, J.Á., Segarra, E., García-Granada, F., Sanchis, E., and Hurtado, L.F. (2022, January 14–16). An Attentional Extractive Summarization Framework. Proceedings of the IberSPEECH, Granada, Spain.10.21437/IberSPEECH.2022-22 ↗
25. Learning-based single-document summarization with compression and anaphoricity constraints10.18653/v1/P16-1188 ↗
26. Narayan, S., Cohen, S.B., and Lapata, M. (November, January 31). Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1206 ↗
27. BERT: Pre-training of deep bidirectional transformers for language understanding
28. Paulus, R., Xiong, C., and Socher, R. (May, January 30). A deep reinforced model for abstractive summarization. Proceedings of the 6th International Conference on Learning Representations ICLR, Vancouver, BC, Canada.
29. Simple statistical gradient-following algorithms for connectionist reinforcement learning10.1007/BF00992696 ↗
30. Mendes, A., Narayan, S., Miranda, S., Marinho, Z., Martins, A.F.T., and Cohen, S.B. (2019, January 2–7). Jointly extracting and compressing documents with summary state representations. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA. Long and Short Papers.10.18653/v1/N19-1397 ↗
31. Get to the point: Summarization with pointer-generator networks10.18653/v1/P17-1099 ↗
32. Ive, J., Madhyastha, P., and Specia, L. (2019, January 3–7). Deep copycat networks for text-to-text generation. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.10.18653/v1/D19-1318 ↗
33. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
34. Wiegreffe, S., and Pinter, Y. (2019, January 3–7). Attention is not not explanation. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.10.18653/v1/D19-1002 ↗
35. Xiao, W., and Carenini, G. (2019, January 3–7). Extractive summarization of long documents by combining global and local context. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.10.18653/v1/D19-1298 ↗
36. Cao, Z., Chen, C., Li, W., Li, S., Wei, F., and Zhou, M. (2016, January 12–17). Tgsum: Build tweet guided multi-document summarization dataset. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.10.1609/aaai.v30i1.10376 ↗
37. Bahdanau, D., Cho, K., and Bengio, Y. (2015, January 7–9). Neural machine translation by jointly learning to align and translate. Proceedings of the 3rd International Conference on Learning Representations, ICLR, San Diego, CA, USA.
38. Luong, T., Pham, H., and Manning, C.D. (2015, January 17–21). Effective approaches to attention-based neural machine translation. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal.10.18653/v1/D15-1166 ↗
39. Bidirectional recurrent neural networks10.1109/78.650093 ↗
40. Ba, L.J., Kiros, J.R., and Hinton, G.E. (2016). Layer normalization. arXiv.
41. Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., and Xiang, B. (2016, January 11–12). Abstractive text summarization using sequence-to-sequence RNNs and beyond. Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, Berlin, Germany.10.18653/v1/K16-1028 ↗
42. Kingma, D.P., and Ba, J. (2015, January 7–9). Adam: A method for stochastic optimization. Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA.
43. Keneshloo, Y., Ramakrishnan, N., and Reddy, C.K. (2019, January 2–4). Deep transfer reinforcement learning for text summarization. Proceedings of the 2019 SIAM International Conference on Data Mining (SDM), Calgary, AB, Canada.10.1137/1.9781611975673.76 ↗
44. Pilault, J., Li, R., Subramanian, S., and Pal, C. (2020, January 16–20). On extractive and abstractive neural document summarization with transformer language models. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Online.10.18653/v1/2020.emnlp-main.748 ↗
45. González, J.Á., Hurtado, L.F., Segarra, E., García-Granada, F., and Sanchis, E. (2019). Summarization of spanish talk shows with Siamese hierarchical attention networks. Appl. Sci., 9.10.3390/app9183836 ↗