FACTMS: Enhancing Multimodal Factual Consistency in Multimodal Summarization
Source abstract
Abstract
Multimodal summarization (MS) generates text summaries from multimedia articles with textual and visual content. Therefore, MS can suffer from the multimodal factual inconsistency problem, where the generated summaries may distort or deviate from both the textual and visual content in the original multimodal input. Existing MS approaches mainly focus on improving the degree of word overlap between output summaries and reference summaries while ignoring the factual inconsistency problem. In this work, we propose a fact-aware multimodal summarization model (FactMS) to improve the multimodal factual consistency of MS. Specifically, we extract the factual information in the textual and visual modalities and remove the text knowledge with low correlation with the visual input. A dual heterogeneous knowledge graph is then proposed to learn the intra-modality factual information in both textual and visual content. By fusing the multimodal factual information in the decoding stage, summaries with better factual consistency are generated. In addition, we propose an automatic method for evaluating visual factual consistency in MS based on the text-image consistency. Extensive experiments conducted on three real-world datasets demonstrate that FactMS achieves an excellent performance of multimodal factual consistency and has a comparable performance in terms of word overlap.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Advanced Text Analysis Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Key Laboratory of Trustworthy Distributed Computing and Service (MoE), Beijing University of Posts and Telecommunications, Beijing 100876, China
School of Computer Science and Engineering, Central South University, Changsha 410083, China
References (50)
Show source-provided references
1. Chen, J., and Zhuge, H. (November, January 31). Abstractive text-image summarization using multi-modal attentional hierarchical rnn. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1438 ↗
2. Li, H., Zhu, J., Liu, T., Zhang, J., and Zong, C. (2018, January 13–19). Multi-modal Sentence Summarization with Modality Attention and Image Filtering. Proceedings of the IJCAI, Stockholm, Sweden.10.24963/ijcai.2018/577 ↗
3. Li, H., Zhu, J., Zhang, J., He, X., and Zong, C. (2020, January 8–13). Multimodal sentence summarization via multimodal selective encoding. Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain.10.18653/v1/2020.coling-main.496 ↗
4. Cao, Z., Wei, F., Li, W., and Li, S. (2018, January 2–7). Faithful to the original: Fact aware neural abstractive summarization. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11912 ↗
5. Kryscinski, W., McCann, B., Xiong, C., and Socher, R. (2020, January 16–20). Evaluating the Factual Consistency of Abstractive Text Summarization. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.10.18653/v1/2020.emnlp-main.750 ↗
6. Huang, L., Wu, L., and Wang, L. (2020, January 5–10). Knowledge Graph-Augmented Abstractive Summarization with Semantic-Driven Cloze Reward. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.457 ↗
7. Zhu, C., Hinthorn, W., Xu, R., Zeng, Q., Zeng, M., Huang, X., and Jiang, M. (2021, January 6–11). Enhancing Factual Consistency of Abstractive Summarization. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online.10.18653/v1/2021.naacl-main.58 ↗
8. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning, PMLR, Online.
9. Text Summarization Techniques: A Brief Survey
10. Studying the effect of syntactic simplification on text summarization10.1080/02564602.2022.2055670 ↗
11. DeepSumm: Exploiting topic models and sequence to sequence networks for extractive text summarization10.1016/j.eswa.2022.118442 ↗
12. Key phrase aware transformer for abstractive summarization10.1016/j.ipm.2022.102913 ↗
13. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2020, January 5–10). BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.703 ↗
14. FAR-ASS: Fact-aware reinforced abstractive sentence summarization10.1016/j.ipm.2020.102478 ↗
15. DOFM: Domain feature miner for robust extractive summarization10.1016/j.ipm.2020.102474 ↗
16. Li, H., Zhu, J., Ma, C., Zhang, J., and Zong, C. (2017, January 9–11). Multi-modal summarization for asynchronous collection of text, image, audio and video. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.10.18653/v1/D17-1114 ↗
17. Palaskar, S., Libovickỳ, J., Gella, S., and Metze, F. (2019). Multimodal abstractive summarization for how2 videos. arXiv.10.18653/v1/P19-1659 ↗
18. Khullar, A., and Arora, U. (2020). MAST: Multimodal Abstractive Summarization with Trimodal Hierarchical Attention. arXiv.10.18653/v1/2020.nlpbt-1.7 ↗
19. Zhang, L., Zhang, X., and Pan, J. (March, January 22). Hierarchical cross-modality semantic correlation learning model for multimodal summarization. Proceedings of the AAAI Conference on Artificial Intelligence, Online.
20. Zhang, Z., Meng, X., Wang, Y., Jiang, X., Liu, Q., and Yang, Z. (March, January 22). Unims: A unified framework for multimodal summarization with knowledge distillation. Proceedings of the AAAI Conference on Artificial Intelligence, Online.
21. Zhang, L., Zhang, X., Guo, Z., and Liu, Z. (2023, January 27–29). CISum: Learning Cross-modality Interaction to Enhance Multimodal Semantic Coverage for Multimodal Summarization. Proceedings of the 2023 SIAM International Conference on Data Mining (SDM), SIAM, Paul Twin Cities, MN, USA.10.1137/1.9781611977653.ch42 ↗
22. Multi-task hierarchical heterogeneous fusion framework for multimodal summarization10.1016/j.ipm.2024.103693 ↗
23. Koncel-Kedziorski, R., Bekal, D., Luan, Y., Lapata, M., and Hajishirzi, H. (2019, January 2–7). Text Generation from Knowledge Graphs with Graph Transformers. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA.
24. Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., and Bengio, Y. (2017). Graph attention networks. arXiv.
25. Sun, Y., Wang, S., Feng, S., Ding, S., Pang, C., Shang, J., Liu, J., Chen, X., Zhao, Y., and Lu, Y. (2021). Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. arXiv.
26. Li, S., He, W., Shi, Y., Jiang, W., Liang, H., Jiang, Y., Zhang, Y., Lyu, Y., and Zhu, Y. (2019, January 9–14). Duie: A large-scale chinese dataset for information extraction. Proceedings of the CCF International Conference on Natural Language Processing and Chinese Computing, Dunhuang, China.10.1007/978-3-030-32236-6_72 ↗
27. Zhang, L., Zhang, X., Zhou, Z., Huang, F., and Li, C. (2024, January 26–27). Reinforced adaptive knowledge learning for multimodal fake news detection. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, BC, Canada.
28. Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., and Zhang, L. (2018, January 18–22). Bottom-up and top-down attention for image captioning and visual question answering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00636 ↗
29. Faster R-CNN: Towards real-time object detection with region proposal networks10.1109/TPAMI.2016.2577031 ↗
30. He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
31. Dense Multi-Agent Reinforcement Learning Aided Multi-UAV Information Coverage for Vehicular Networks10.1109/JIOT.2024.3367005 ↗
32. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
33. Tan, H., and Bansal, M. (2019). Lxmert: Learning cross-modality encoder representations from transformers. arXiv.10.18653/v1/D19-1514 ↗
34. Zhang, L., Zhang, X., Li, C., Zhou, Z., Liu, J., Huang, F., and Zhang, X. (November, January 28). Mitigating Social Hazards: Early Detection of Fake News via Diffusion-Guided Propagation Path Generation. Proceedings of the ACM Multimedia, Melbourne, Australia.
35. Dou, Z.Y., Liu, P., Hayashi, H., Jiang, Z., and Neubig, G. (2021, January 6–11). Gsum: A general framework for guided neural abstractive summarization. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online.10.18653/v1/2021.naacl-main.384 ↗
36. Fu, X., Wang, J., and Yang, Z. (2020). Multi-modal summarization for video-containing documents. arXiv.
37. Hua, L., Wan, X., and Li, L. (2017, January 8–12). Overview of the NLPCC 2017 shared task: Single document summarization. Proceedings of the National CCF Conference on Natural Language Processing and Chinese Computing, Dalian, China.10.1007/978-3-319-73618-1_84 ↗
38. Sun, M., Li, J., Guo, Z., Yu, Z., Zheng, Y., Si, X., and Liu, Z. (2016). Thuctc: An efficient chinese text classifier. GitHub Repos.
39. Early Detection of Multimodal Fake News via Reinforced Propagation Path Generation10.1109/TKDE.2024.3496701 ↗
40. Lin, C.Y. (2004, January 25–26). Rouge: A package for automatic evaluation of summaries. Proceedings of the Text Summarization Branches Out, Barcelona, Spain.
41. Papineni, K., Roukos, S., Ward, T., and Zhu, W.J. (2002, January 6–15). Bleu: A method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA.10.3115/1073083.1073135 ↗
42. Zhu, J., Li, H., Liu, T., Zhou, Y., Zhang, J., and Zong, C. (November, January 31). MSMO: Multimodal summarization with multimodal output. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1448 ↗
43. Zhu, J., Zhou, Y., Zhang, J., Li, H., Zong, C., and Li, C. (2020, January 7–12). Multimodal summarization with guidance of multimodal reference. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i05.6525 ↗
44. Li, M., Chen, X., Gao, S., Chan, Z., Zhao, D., and Yan, R. (2020). VMSMO: Learning to Generate Multimodal Summary for Video-based News Articles. arXiv.10.18653/v1/2020.emnlp-main.752 ↗
45. Luong, M.T., Pham, H., and Manning, C.D. (2015). Effective approaches to attention-based neural machine translation. arXiv.10.18653/v1/D15-1166 ↗
46. See, A., Liu, P.J., and Manning, C.D. (2017). Get to the point: Summarization with pointer-generator networks. arXiv.10.18653/v1/P17-1099 ↗
47. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
48. Kingma, D.P., and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv.
49. Mihalcea, R., and Tarau, P. (2004, January 25–26). Textrank: Bringing order into text. Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, Barcelona, Spain.
50. UzZaman, N., Bigham, J.P., and Allen, J.F. (2011, January 13–16). Multimodal summarization of complex sentences. Proceedings of the 16th International Conference on Intelligent User Interfaces, Palo Alto, CA, USA.10.1145/1943403.1943412 ↗