ACR: Adaptive Confidence Re-Scoring for Reliable Answer Selection Among Multiple Candidates
Source abstract
Abstract
With the improved reasoning capabilities of large language models (LLMs), their applications have rapidly expanded across a wide range of tasks. In recent question answering tasks, performance gains have been achieved through Self-Consistency, where LLMs generate multiple reasoning paths and determine the final answer via majority voting. However, this approach can fail when the correct answer is generated but does not appear frequently enough to be selected, highlighting its vulnerability to inconsistent generations. To address this, we propose Adaptive Confidence Re-scoring (ACR)—a method that adaptively evaluates and re-scores candidate answers to select the most trustworthy one when LLMs fail to generate consistent reasoning. Experiments on arithmetic and logical reasoning benchmarks show that ACR maintains or improves answer accuracy while significantly reducing inference cost. Compared to existing verification methods such as FOBAR, ACR reduces the number of inference calls by up to 95%, while improving inference efficiency—measured as accuracy gain per inference call—by a factor of 2× to 17×, depending on the dataset and model.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Expert finding and Q&A systems · Spam and Phishing Detection
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Artificial Intelligence, Hanyang University, Seoul 04763, Republic of Korea
Department of Computer Science, Hanyang University, Seoul 04763, Republic of Korea
Funding
Institute of Information and communications Technology Planning and evaluation (IITP) · National Research Foundation of Korea (NRF)
References (32)
Show source-provided references
1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., and Altman, S. (2025, August 01). GPT-4 Technical Report. Available online: http://arxiv.org/abs/2303.08774.
2. Anthropic (2025, July 26). claude-3-7-sonnet. Available online: https://www.anthropic.com/news/claude-3-7-sonnet.
3. Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., and Rosen, E. (2025, August 01). Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. Available online: http://arxiv.org/abs/2507.06261.
4. Language Models are Few-Shot Learners
5. Rogers, A., Boyd-Graber, J., and Okazaki, N. (2023). Few-shot In-context Learning on Knowledge Base Question Answering. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, Toronto, ON, Canada, 9–14 July 2023 (Volume 1: Long Papers), Association for Computational Linguistics.
6. Coda-Forno, J., Binz, M., Akata, Z., Botvinick, M., Wang, J.X., and Schulz, E. (2023, January 10–16). Meta-in-context learning in large language models. Proceedings of the Thirty-Seventh Conference on Neural Information Processing Systems, New Orleans, LA, USA.
7. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
8. Large Language Models are Zero-Shot Reasoners
9. Fu, Y., Peng, H., Sabharwal, A., Clark, P., and Khot, T. (2023, January 1–5). Complexity-Based Prompting for Multi-step Reasoning. Proceedings of the Eleventh International Conference on Learning Representations, Kigali, Rwanda.
10. Ku, L.W., Martins, A., and Srikumar, V. (2024). ERA-CoT: Improving Chain-of-Thought through Entity Relationship Analysis. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, Bangkok, Thailand, 11–16 August 2024 (Volume 1: Long Papers), Association for Computational Linguistics.
11. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. (2023, January 1–5). Self-Consistency Improves Chain of Thought Reasoning in Language Models. Proceedings of the International Conference on Learning Representations (ICLR), Kigali, Rwanda.
12. Li, Y., Yuan, P., Feng, S., Pan, B., Wang, X., Sun, B., Wang, H., and Li, K. (2024, January 7–11). Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoning. Proceedings of the Twelfth International Conference on Learning Representations, Vienna, Austria.
13. Bouamor, H., Pino, J., and Bali, K. (2023). Large Language Models are Better Reasoners with Self-Verification. Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, 6–10 December 2023, Association for Computational Linguistics.
14. Ku, L.W., Martins, A., and Srikumar, V. (2024). Forward-Backward Reasoning in Large Language Models for Mathematical Verification. Proceedings of the Findings of the Association for Computational Linguistics: ACL 2024, Bangkok, Thailand, 11–16 August 2024, Association for Computational Linguistics.
15. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., and Nakano, R. (2025, August 01). Training Verifiers to Solve Math Word Problems. Available online: http://arxiv.org/abs/2110.14168.
16. Lu, P., Qiu, L., Chang, K.W., Wu, Y.N., Zhu, S.C., Rajpurohit, T., Clark, P., and Kalyan, A. (2023, January 1–5). Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning. Proceedings of the International Conference on Learning Representations (ICLR), Kigali, Rwanda.
17. Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. (2025, August 01). Let’s Verify Step by Step. Available online: http://arxiv.org/abs/2305.20050.
18. Rogers, A., Boyd-Graber, J., and Okazaki, N. (2023). Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. Proceedings of the Findings of the Association for Computational Linguistics: ACL 2023, Toronto, ON, Canada, 9–14 July 2023, Association for Computational Linguistics.
19. Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., and Wei, H. (2025, August 01). Qwen2.5 Technical Report. Available online: http://arxiv.org/abs/2412.15115.
20. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., and Lv, C. (2025, August 01). Qwen3 Technical Report. Available online: http://arxiv.org/abs/2505.09388.
21. Mistral-AI (2025, July 26). Mistral-Small-3.1. Available online: https://mistral.ai/news/mistral-small-3.10.5840/mr201816 ↗
22. Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lillicrap, T., Alayrac, J.b., Soricut, R., Lazaridou, A., Firat, O., and Schrittwieser, J. (2024). Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv.
23. Learning to summarize with human feedback
24. Sessa, P.G., Dadashi, R., Hussenot, L., Ferret, J., Vieillard, N., Ramé, A., Shariari, B., Perrin, S., Friesen, A., and Cideron, G. (2025, August 01). BOND: Aligning LLMs with Best-of-N Distillation. Available online: http://arxiv.org/abs/2407.14622.
25. Ku, L.W., Martins, A., and Srikumar, V. (2024). Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, Bangkok, Thailand, 11–16 August 2024 (Volume 1: Long Papers), Association for Computational Linguistics.
26. Al-Onaizan, Y., Bansal, M., and Chen, Y.N. (2024). Self-Consistency Boosts Calibration for Math Reasoning. Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, FL, USA, 12–16 November 2024, Association for Computational Linguistics.
27. Bouamor, H., Pino, J., and Bali, K. (2023). Let’s Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, 6–10 December 2023, Association for Computational Linguistics.
28. Ku, L.W., Martins, A., and Srikumar, V. (2024). Soft Self-Consistency Improves Language Models Agents. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, Bangkok, Thailand, 11–16 August 2024 (Volume 2: Short Papers), Association for Computational Linguistics.
29. Duh, K., Gomez, H., and Bethard, S. (2024). Paraphrase and Solve: Exploring and Exploiting the Impact of Surface Form on Mathematical Reasoning in Large Language Models. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Mexico City, Mexico, 16–21 June 2024 (Volume 1: Long Papers), Association for Computational Linguistics.
30. Ku, L.W., Martins, A., and Srikumar, V. (2024). Self-Para-Consistency: Improving Reasoning Tasks at Low Cost for Large Language Models. Proceedings of the Findings of the Association for Computational Linguistics: ACL 2024, Bangkok, Thailand, 11–16 August 2024, Association for Computational Linguistics.
31. Vanschoren, J., and Yeung, S. (2021;, January 6–14). Measuring Mathematical Problem Solving with the MATH Dataset. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, Virtual.
32. Zheng, C., Zhou, H., Meng, F., Zhou, J., and Huang, M. (2024, January 7–11). Large Language Models Are Not Robust Multiple Choice Selectors. Proceedings of the Twelfth International Conference on Learning Representations, Vienna, Austria.