Testing Explainability of Chain of Thought for Large Language Models
Source abstract
Abstract
Large Language Models (LLMs) have demonstrated superior abilities in complex tasks such as text generation, reasoning, and question answering. However, the explainability of LLMs becomes weak as the parameters and complexity of LLMs increase. Chains of Thought (CoTs) guide the model to perform step-by-step reasoning and effectively enhance its reasoning ability. The multi-step rationales verbalized in a CoT are widely regarded as the explanation of the model itself. This paper proposes an automated approach to testing the behavioral sensitivity of responses to self-cited evidence in CoTs from sufficiency and necessity perspectives under context intervention. Specifically, we intervene in the reasoning chain by changing the input context and measure the behavioral consistency as a proxy for the faithfulness of the CoT. We test the CoT rationales of mainstream open-source LLMs on multi-hop question-answering tasks. The experimental results show that the self-stated reasoning chain is insufficient and unnecessary. The CoT cannot fully explain the behavior of LLMs.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Explainable Artificial Intelligence (XAI) · Multimodal Machine Learning Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science, Wuhan University, Wuhan 430072, China
Funding
National Natural Science Foundation of China
References (41)
Show source-provided references
1. DARPA’s Explainable Artificial Intelligence (XAI) Program
2. Definitions, methods, and applications in interpretable machine learning10.1073/pnas.1900654116 ↗
3. Guyon, I., von Luxburg, U., Bengio, S., Wallach, H.M., Fergus, R., Vishwanathan, S.V.N., and Garnett, R. (2017). A Unified Approach to Interpreting Model Predictions. Advances in Neural Information Processing Systems 30, Proceedings of the Annual Conference on Neural Information Processing Systems 2017, Long Beach, CA, USA, 4–9 December 2017, NIPS Foundation.
4. Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (2022). Large Language Models are Zero-Shot Reasoners. Advances in Neural Information Processing Systems 35, Proceedings of the Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, 28 November–9 December 2022, NIPS Foundation.
5. Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (2023). Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. Advances in Neural Information Processing Systems 36, Proceedings of the Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, 10–16 December 2023, NIPS Foundation.
6. Doshi-Velez, F., and Kim, B. (2017). Towards A Rigorous Science of Interpretable Machine Learning. arXiv.
7. Notions of explainability and evaluation approaches for explainable artificial intelligence10.1016/j.inffus.2021.05.009 ↗
8. A systematic review and taxonomy of explanations in decision support and recommender systems10.1007/s11257-017-9195-0 ↗
9. Chen, Y., Zhong, R., Ri, N., Zhao, C., He, H., Steinhardt, J., Yu, Z., and McKeown, K.R. (2024, January 21–27). Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations. Proceedings of the Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria.
10. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI10.1016/j.inffus.2019.12.012 ↗
11. Silva, L., Metrôlho, J.C., and Ribeiro, F.R. (2024). Efficient Data Exchange between WebAssembly Modules. Future Internet, 16.10.3390/fi16090341 ↗
12. Biran, O., and Cotton, C. (2017, January 20). Explanation and justification in machine learning: A survey. Proceedings of the IJCAI-17 Workshop on Explainable AI (XAI), Melbourne, Australia.
13. Lee, D.D., Sugiyama, M., von Luxburg, U., Guyon, I., and Garnett, R. (2016). Examples are not enough, learn to criticize! Criticism for Interpretability. Advances in Neural Information Processing Systems 29, Proceedings of the Annual Conference on Neural Information Processing Systems 2016, Barcelona, Spain, 5–10 December 2016, NIPS Foundation.
14. Explainability for Large Language Models: A Survey
15. Explain Yourself! Leveraging Language Models for Commonsense Reasoning
16. Bills, S., Cammarata, N., Mossing, D., Tillman, H., Gao, L., Goh, G., Sutskever, I., Leike, J., Wu, J., and Saunders, W. (2026, March 16). Language Models Can Explain Neurons in Language Models. Available online: https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html.
17. A Mathematical Framework for Transformer Circuits
18. Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (2022). Locating and Editing Factual Associations in GPT. Advances in Neural Information Processing Systems 35, Proceedings of the Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, 28 November–9 December 2022, NIPS Foundation.
19. Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J.R. (2020). ERASER: A Benchmark to Evaluate Rationalized NLP Models. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Association for Computational Linguistics.
20. Rogers, A., Boyd-Graber, J.L., and Okazaki, N. (2023). Faithfulness Tests for Natural Language Explanations. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, (Volume 2: Short Papers), ACL 2023, Association for Computational Linguistics.
21. Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Advances in Neural Information Processing Systems 35, Proceedings of the Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, 28 November–9 December 2022, NIPS Foundation.
22. Wang, X., Wei, J., Schuurmans, D., Le, Q.V., Chi, E.H., Narang, S., Chowdhery, A., and Zhou, D. (2023, January 1–5). Self-Consistency Improves Chain of Thought Reasoning in Language Models. Proceedings of the Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda.
23. Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., and Bi, X. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv.
24. Faithful Chain-of-Thought Reasoning
25. Knowledge Editing for Large Language Models: A Survey
26. Rogers, A., Boyd-Graber, J.L., and Okazaki, N. (2023). Teaching Small Language Models to Reason. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), ACL 2023, Association for Computational Linguistics.
27. MuSiQue: Multihop Questions via Single-hop Question Composition10.1162/tacl_a_00475 ↗
28. Bouamor, H., Pino, J., and Bali, K. (2023). MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Association for Computational Linguistics.
29. Can ChatGPT Replace Traditional KBQA Models? An In-Depth Analysis of the Question Answering Performance of the GPT LLM Family
30. Bouamor, H., Pino, J., and Bali, K. (2023). Sources of Hallucination by Large Language Models on Inference Tasks. Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2023, Association for Computational Linguistics.
31. Debt-Prone Bugs: Technical Debt in Software Maintenance
32. Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J. (2018). HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics.
33. Globersons, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J.M., and Zhang, C. (2024). HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models. Advances in Neural Information Processing Systems 38, Proceedings of the Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, 10–15 December 2024, NIPS Foundation.
34. Ren, Z., Sun, S., Xuan, J., Li, X., Zhou, Z., and Jiang, H. (2022, January 25–27). Automated Patching for Unreproducible Builds. Proceedings of the 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA.10.1145/3510003.3510102 ↗
35. Yang, F., Xin, Q., Ren, Z., and Xuan, J. (2025, January 27–28). Kotsuite: Unit Test Generation for Kotlin Programs in Android Applications. Proceedings of the 33rd IEEE/ACM International Conference on Program Comprehension, ICPC@ICSE 2025, Ottawa, ON, Canada.10.1109/ICPC66645.2025.00032 ↗
36. A Developer-Focused Genetic Algorithm for IoT Application Placement in the Computing Continuum10.1109/TSC.2025.3556641 ↗
37. Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 13–17). “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.10.1145/2939672.2939778 ↗
38. A bi-directional adversarial explainability for decision support10.1007/s42454-021-00031-5 ↗
39. Shah, S.S., and Sheppard, J.W. (2020, January 19–24). Evaluating Explanations of Convolutional Neural Network Image Classifications. Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK.10.1109/IJCNN48605.2020.9207129 ↗
40. Warnecke, A., Arp, D., Wressnegger, C., and Rieck, K. (2020, January 7–11). Evaluating Explanation Methods for Deep Learning in Security. Proceedings of the 2020 IEEE European Symposium on Security and Privacy (Euro S&P), Genoa, Italy.10.1109/EuroSP48549.2020.00018 ↗
41. Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., and Garnett, R. (2019). A Benchmark for Interpretability Methods in Deep Neural Networks. Advances in Neural Information Processing Systems, Curran Associates, Inc.