Mind the Link: Discourse Link-Aware Hallucination Detection in Summarization
Source abstract
Abstract
Recent studies on detecting hallucinations in summaries follow a method of decomposing summaries into atomic content units (ACUs) and then determining whether each unit logically matches the document text based on natural language inference. However, this fails to consider discourse link relations such as temporal order, causality, and purpose, leading to the inability to detect conflicts in semantic connections between individual summary ACUs, even when the conflicts are present in the document. To overcome this limitation, this study proposes a method of extracting Discourse Link-Aware Content Unit (DL-ACU) by converting the summary into an Abstract Meaning Representation (AMR) graph and structuring the discourse link relations between ACUs. Additionally, to align summary ACUs with corresponding document information in a fine-grained manner, we propose a Selective Document-Atomic Content Unit (SD-ACU). For each summary ACU, the SD-ACU retrieves only the most relevant document sentences and then decomposes them into document ACUs. Applying the DL-ACU module to existing hallucination detection systems such as FIZZ and FENICE reduces the error rate of discourse link errors on FRANK. When both modules are combined, the system improves balanced accuracy and ROC-AUC across major benchmarks. This suggests the proposed method effectively captures discourse link errors while enabling ACU-to-ACU alignment.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Text Analysis Techniques · Complex Network Analysis Techniques · Topic Modeling
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Intelligence and Convergence, Hanyang University, Seoul 04763, Republic of Korea
Meta, Ads Privacy ML Infra, New York, NY 10003, USA
Department of Computer Science, Hanyang University, Seoul 04763, Republic of Korea
Funding
Institute of Information and communications Technology Planning and evaluation · National Research Foundation of Korea
References (53)
Show source-provided references
1. Liu, Y., Fabbri, A., Liu, P., Zhao, Y., Nan, L., Han, R., Han, S., Joty, S., Wu, C.S., and Xiong, C. (2023, January 9–14). Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, ON, Canada.10.18653/v1/2023.acl-long.228 ↗
2. Yang, J., Yoon, S., Kim, B., and Lee, H. (2024, January 12–16). FIZZ: Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Document. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, FL, USA.10.18653/v1/2024.emnlp-main.3 ↗
3. Scirè, A., Ghonim, K., and Navigli, R. (2024, January 11–16). FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction. Proceedings of the Findings of the Association for Computational Linguistics: ACL 2024, Bangkok, Thailand.10.18653/v1/2024.findings-acl.841 ↗
4. Zhang, H., Xu, Y., and Perez-Beltrachini, L. (2024, January 17–22). Fine-Grained Natural Language Inference Based Faithfulness Evaluation for Diverse Summarisation Tasks. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), St. Julian’s, Malta.10.18653/v1/2024.eacl-long.102 ↗
5. Pagnoni, A., Balachandran, V., and Tsvetkov, Y. (2021, January 6–11). Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online.10.18653/v1/2021.naacl-main.383 ↗
6. Luo, G., Fan, W., Li, M., He, Y., Yang, Y., and Bao, F. (2024, January 17–22). On the Intractability to Synthesize Factual Inconsistencies in Summarization. Proceedings of the Findings of the Association for Computational Linguistics: EACL 2024, St. Julian’s, Malta.
7. Qiu, H., Huang, K.H., Qu, J., and Peng, N. (2024, January 16–21). AMRFact: Enhancing Summarization Factuality Evaluation with AMR-Driven Negative Samples Generation. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Mexico City, Mexico.10.18653/v1/2024.naacl-long.33 ↗
8. SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization10.1162/tacl_a_00453 ↗
9. Zha, Y., Yang, Y., Li, R., and Hu, Z. (2023, January 9–14). AlignScore: Evaluating Factual Consistency with A Unified Alignment Function. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, ON, Canada.10.18653/v1/2023.acl-long.634 ↗
10. Stacey, J., Minervini, P., Dubossarsky, H., Camburu, O.M., and Rei, M. (2024, January 12–16). Atomic Inference for NLI with Generated Facts as Atoms. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, FL, USA.10.18653/v1/2024.emnlp-main.569 ↗
11. Mitra, A., Corro, L.D., Mahajan, S., Codas, A., Simoes, C., Agarwal, S., Chen, X., Razdaibiedina, A., Jones, E., and Aggarwal, K. (2023). Orca 2: Teaching Small Language Models How to Reason. arXiv.
12. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
13. Banarescu, L., Bonial, C., Cai, S., Georgescu, M., Griffitt, K., Hermjakob, U., Knight, K., Koehn, P., Palmer, M., and Schneider, N. (2013, January 8–9). Abstract Meaning Representation for Sembanking. Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, Sofia, Bulgaria.
14. MENLI: Robust Evaluation Metrics from Natural Language Inference10.1162/tacl_a_00576 ↗
15. Kryscinski, W., McCann, B., Xiong, C., and Socher, R. (2020, January 16–20). Evaluating the Factual Consistency of Abstractive Text Summarization. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.10.18653/v1/2020.emnlp-main.750 ↗
16. Bao, F.S., Li, M., Qu, R., Luo, G., Wan, E., Tang, Y., Fan, W., Tamber, M.S., Kazi, S., and Sourabh, V. (May, January 29). FaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMs. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), Albuquerque, Mexico.10.18653/v1/2025.naacl-short.38 ↗
17. Scialom, T., Dray, P.A., Lamprier, S., Piwowarski, B., Staiano, J., Wang, A., and Gallinari, P. (2021, January 7–11). QuestEval: Summarization Asks for Fact-based Evaluation. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic.10.18653/v1/2021.emnlp-main.529 ↗
18. Fabbri, A., Wu, C.S., Liu, W., and Xiong, C. (2022, January 10–15). QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, WA, USA.10.18653/v1/2022.naacl-main.187 ↗
19. Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., and Zhu, C. (2023, January 6–10). G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore.10.18653/v1/2023.emnlp-main.153 ↗
20. Wan, D., Sinha, K., Iyer, S., Celikyilmaz, A., Bansal, M., and Pasunuru, R. (2024, January 11–16). ACUEval: Fine-grained Hallucination Evaluation and Correction for Abstractive Summarization. Proceedings of the Findings of the Association for Computational Linguistics: ACL 2024, Bangkok, Thailand.10.18653/v1/2024.findings-acl.597 ↗
21. Ribeiro, L.F.R., Liu, M., Gurevych, I., Dreyer, M., and Bansal, M. (2022, January 10–15). FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, WA, USA.10.18653/v1/2022.naacl-main.236 ↗
22. Goyal, T., and Durrett, G. (2020, January 16–20). Evaluating Factuality in Generation with Dependency-level Entailment. Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2020, Online.10.18653/v1/2020.findings-emnlp.322 ↗
23. Nenkova, A., and Passonneau, R. (2004, January 2–7). Evaluating Content Selection in Summarization: The Pyramid Method. Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, Boston, MA, USA.
24. Goodman, M.W. (2020, January 5–10). Penman: An Open-Source Library and Tool for AMR Graphs. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Online.10.18653/v1/2020.acl-demos.35 ↗
25. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2020, January 5–10). BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.703 ↗
26. Wein, S., and Opitz, J. (2024, January 12–16). A Survey of AMR Applications. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, FL, USA.10.18653/v1/2024.emnlp-main.390 ↗
27. Nawrath, M., Nowak, A., Ratz, T., Walenta, D., Opitz, J., Ribeiro, L., Sedoc, J., Deutsch, D., Mille, S., and Liu, Y. (2024, January 16–21). On the Role of Summary Content Units in Text Summarization Evaluation. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), Mexico City, Mexico.10.18653/v1/2024.naacl-short.25 ↗
28. Martinelli, G., Barba, E., and Navigli, R. (2024, January 11–16). Maverick: Efficient and Accurate Coreference Resolution Defying Recent Trends. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand.10.18653/v1/2024.acl-long.722 ↗
29. Kamoi, R., Goyal, T., Diego Rodriguez, J., and Durrett, G. (2023, January 6–10). WiCE: Real-World Entailment for Claims in Wikipedia. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore.10.18653/v1/2023.emnlp-main.470 ↗
30. Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., and Artzi, Y. (2020, January 26–30). BERTScore: Evaluating Text Generation with BERT. Proceedings of the 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia.
31. Schuster, T., Fisch, A., and Barzilay, R. (2021, January 6–11). Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online.10.18653/v1/2021.naacl-main.52 ↗
32. Pradhan, S., Moschitti, A., Xue, N., Uryupina, O., and Zhang, Y. (2012, January 12–14). CoNLL-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes. Proceedings of the Joint Conference on EMNLP and CoNLL, Shared Task, Jeju Island, Republic of Korea.
33. Tang, L., Goyal, T., Fabbri, A., Laban, P., Xu, J., Yavuz, S., Kryscinski, W., Rousseau, J., and Durrett, G. (2023, January 9–14). Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, ON, Canada.10.18653/v1/2023.acl-long.650 ↗
34. SummEval: Re-evaluating Summarization Evaluation10.1162/tacl_a_00373 ↗
35. Cao, S., and Wang, L. (2021, January 7–11). CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic.10.18653/v1/2021.emnlp-main.532 ↗
36. Maynez, J., Narayan, S., Bohnet, B., and McDonald, R. (2020, January 5–10). On Faithfulness and Factuality in Abstractive Summarization. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.173 ↗
37. Huang, D., Cui, L., Yang, S., Bao, G., Wang, K., Xie, J., and Zhang, Y. (2020, January 16–20). What Have We Achieved on Text Summarization?. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.10.18653/v1/2020.emnlp-main.33 ↗
38. Goyal, T., and Durrett, G. (2021, January 6–11). Annotating and Modeling Fine-grained Factuality in Summarization. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online.10.18653/v1/2021.naacl-main.114 ↗
39. Cao, M., Dong, Y., and Cheung, J. (2022, January 22–27). Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarization. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland.10.18653/v1/2022.acl-long.236 ↗
40. Wang, A., Cho, K., and Lewis, M. (2020, January 5–10). Asking and Answering Questions to Evaluate the Factual Consistency of Summaries. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.450 ↗
41. See, A., Liu, P.J., and Manning, C.D. (August, January 30). Get To The Point: Summarization with Pointer-Generator Networks. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, BC, Canada.
42. Gehrmann, S., Deng, Y., and Rush, A. (November, January 31). Bottom-Up Abstractive Summarization. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1443 ↗
43. Liu, Y., and Lapata, M. (2019, January 3–7). Text Summarization with Pretrained Encoders. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.10.18653/v1/D19-1387 ↗
44. Language Models are Unsupervised Multitask Learners
45. Zhang, J., Zhao, Y., Saleh, M., and Liu, P.J. (2020, January 13–18). PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization. Proceedings of the 37th International Conference on Machine Learning, Online.
46. Hermann, K.M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., and Blunsom, P. (2015, January 7–12). Teaching Machines to Read and Comprehend. Proceedings of the 28th International Conference on Neural Information Processing Systems (NeurIPS), Montreal, QC, Canada.
47. Narayan, S., Cohen, S.B., and Lapata, M. (November, January 31). Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1206 ↗
48. Falke, T., Ribeiro, L.F.R., Utama, P.A., Dagan, I., and Gurevych, I. (August, January 28). Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
49. Adams, G., Nguyen, B., Smith, J., Xia, Y., Xie, S., Ostropolets, A., Deb, B., Chen, Y.J., Naumann, T., and Elhadad, N. (2023, January 9–14). What are the Desired Characteristics of Calibration Sets? Identifying Correlates on Long Form Scientific Summarization. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, ON, Canada.10.18653/v1/2023.acl-long.587 ↗
50. Cohan, A., Dernoncourt, F., Kim, D.S., Bui, T., Kim, S., Chang, W., and Goharian, N. (2018, January 1–6). A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), New Orleans, LA, USA.10.18653/v1/N18-2097 ↗
51. Huang, L., Cao, S., Parulian, N., Ji, H., and Wang, L. (2021, January 6–11). Efficient Attentions for Long Document Summarization. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online.10.18653/v1/2021.naacl-main.112 ↗
52. Fabbri, A., Li, I., She, T., Li, S., and Radev, D. (August, January 28). Multi-News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
53. Zhong, M., Yin, D., Yu, T., Zaidi, A., Mutuma, M., Jha, R., Awadallah, A.H., Celikyilmaz, A., Liu, Y., and Qiu, X. (2021, January 6–11). QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online.10.18653/v1/2021.naacl-main.472 ↗