An Adaptive Multi-Source Retrieval-Augmented Generation Framework Integrating Query Complexity Awareness and Confidence-Aware Fusion
Source abstract
Abstract
Retrieval-Augmented Generation (RAG) has been observed to encounter challenges in heterogeneous query scenarios characterised by varying evidence requirements and reasoning depths. In order to address this limitation, the present paper puts forward a proposal for an Adaptive Multi-Source RAG framework (AMSRAG) that integrates query complexity awareness with confidence-aware fusion. The framework performs query complexity classification with a pretrained language model, calibrates the classification confidence to guide the dynamic scheduling of retrieval paths and the adjustment of fusion weights, and enables a controllable balance between answer quality and retrieval efficiency through hierarchical path selection and cross-source weighting. The experiments conducted on multiple open-domain question-answering datasets demonstrate that the query complexity classifier achieves an accuracy of 85.9% and a Macro-F1 score of 85.4%. These outcomes indicate the potential for the classifier to generate a reliable decision signal, which can subsequently be utilised to guide the process of adaptive retrieval and fusion. The proposed framework demonstrates a marked improvement in terms of both answer accuracy and retrieval relevance when compared to the fixed-pipeline RAG. In scenarios involving high-confidence queries, the system has been shown to effectively avoid redundant retrieval, thereby reducing the average number of retrievals. In instances of low-confidence complex queries, the system has been shown to enhance evidence coverage and completeness of answers through multi-source retrieval and confidence-weighted fusion. This study proposes a novel methodology for enhancing the adaptability and resource efficiency of RAG systems in response to heterogeneous query conditions.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Information Retrieval and Search Behavior · Topic Modeling · Advanced Graph Neural Networks
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Artificial Intelligence, China University of Geosciences Beijing, Beijing 100083, China
Funding
National College Students Innovation and Entrepreneurship Training Program of China · Beijing Higher Education Society Approved Research Projects · Undergraduate Education Quality Improvement Plan Construction Project of China University of Geosciences
References (28)
Show source-provided references
1. Retrieval-augmented generation for knowledge-intensive NLP tasks
2. Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M.W. (2020, January 13–18). REALM: Retrieval-augmented language model pre-training. Proceedings of the 37th International Conference on Machine Learning, Vienna, Austria.
3. Atlas: Few-shot learning with retrieval-augmented language models
4. Izacard, G., and Grave, E. (2021, January 19–23). Leveraging passage retrieval with generative models for open-domain question answering. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, Online.10.18653/v1/2021.eacl-main.74 ↗
5. Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Van Den Driessche, G.B., Lespiau, J.B., Damoc, B., and Clark, A. (2022, January 17–23). Improving language models by retrieving from trillions of tokens. Proceedings of the 39th International Conference on Machine Learning, Baltimore, MD, USA.
6. Edge, D., Trinh, H., Newman Cheng, J.B., Chao, A., Mody, A., Truitt, S., Metropolitansky, D., Ness, R.O., and Larson, J. (2024). From local to global: A graph RAG approach to query-focused summarization. arXiv.
7. Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.T. (2020, January 16–20). Dense passage retrieval for open-domain question answering. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Online.10.18653/v1/2020.emnlp-main.550 ↗
8. Asai, A., Wu, Z., Wang, Y., Sil, A., and Hajishirzi, H. (2024, January 7–11). Self-RAG: Learning to retrieve, generate, and critique through self-reflection. Proceedings of the 12th International Conference on Learning Representations, Vienna, Austria.
9. Jeong, S., Baek, J., Cho, S., Hwang, S.J., and Park, J.C. (2024, January 16–21). Adaptive-RAG: Learning to adapt retrieval-augmented large language models through question complexity. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics, Mexico City, Mexico.10.18653/v1/2024.naacl-long.389 ↗
10. Fox, E.A., and Shaw, J.A. (September, January 31). Combination of multiple searches. Proceedings of the Second Text REtrieval Conference (TREC-2), Gaithersburg, MD, USA.
11. Cormack, G.V., Clarke, C.L.A., and Buettcher, S. (2009, January 19–23). Reciprocal rank fusion outperforms Condorcet and individual rank learning methods. Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, Boston, MA, USA.10.1145/1571941.1572114 ↗
12. Adapting boosting for information retrieval measures10.1007/s10791-009-9112-1 ↗
13. Cao, Y., Xu, J., Liu, T.Y., Li, H., Huang, Y., and Hon, H.W. (2006, January 6–11). Adapting ranking SVM to document retrieval. Proceedings of the 29th International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle, WA, USA.10.1145/1148170.1148205 ↗
14. Wang, X., Macdonald, C., Tonellotto, N., and Ounis, I. (2021, January 11–15). Pseudo-relevance feedback for multiple representation dense retrieval. Proceedings of the 2021 ACM SIGIR International Conference on the Theory of Information Retrieval, Virtual Event.10.1145/3471158.3472250 ↗
15. Carbonell, J., and Goldstein, J. (1998, January 24–28). The use of MMR, diversity-based reranking for reordering documents and producing summaries. Proceedings of the 21st International ACM SIGIR Conference on Research and Development in Information Retrieval, Melbourne, Australia.10.1145/290941.291025 ↗
16. Guo, C., Pleiss, G., Sun, Y., and Weinberger, K.Q. (2017, January 6–11). On calibration of modern neural networks. Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia.
17. Classifier calibration: A survey on how to assess and improve predicted class probabilities10.1007/s10994-023-06336-7 ↗
18. Rajaraman, S., Ganesan, P., and Antani, S. (2022). Deep learning model calibration for improving performance in class-imbalanced medical image classification tasks. PLoS ONE, 17.10.1371/journal.pone.0262838 ↗
19. A differentiable and uncertainty-aware mutual information regularizer for bias mitigation10.1016/j.neucom.2025.132498 ↗
20. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2–7). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, Minneapolis, MN, USA.
21. Warner, B., Chaffin, A., Clavié, B., Weller, O., Hallström, O., Taghadouini, S., Gallagher, A., Biswas, R., Ladhak, F., and Aarsen, T. (August, January 27). Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory-efficient, and long-context finetuning and inference. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025), Vienna, Austria.10.18653/v1/2025.acl-long.127 ↗
22. Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., and Nguyen, T. (2016, January 5–10). MS MARCO: A human generated machine reading comprehension dataset. Proceedings of the Neural Information Processing Systems Workshop, Barcelona, Spain.
23. Yang, Z., Qi, P., Zhang, S., Cohen, W.W., Salakhutdinov, R., and Manning, C.D. (November, January 31). HotpotQA: A dataset for diverse, explainable multi-hop question answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1259 ↗
24. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., and Altman, S. (2023). GPT-4 technical report. arXiv.
25. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. (2023, January 1–5). Self-consistency improves chain-of-thought reasoning in language models. Proceedings of the 11th International Conference on Learning Representations, Kigali, Rwanda.
26. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A robustly optimized BERT pretraining approach. arXiv.
27. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. (2016, January 1–5). SQuAD: 100,000+ questions for machine comprehension of text. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.10.18653/v1/D16-1264 ↗
28. Karmaker, S.K., Sondhi, P., and Zhai, C. (2020, January 19–23). Empirical analysis of impact of query-specific customization of nDCG: A case-study with learning-to-rank methods. Proceedings of the 29th ACM International Conference on Information and Knowledge Management, Galway, Ireland.10.1145/3340531.3417454 ↗