Layered Query Retrieval: An Adaptive Framework for Retrieval-Augmented Generation in Complex Question Answering for Large Language Models
Source abstract
Abstract
Retrieval-augmented generation (RAG) addresses the problem of knowledge cutoff and overcomes the inherent limitations of pre-trained language models by retrieving relevant information in real time. However, challenges related to efficiency and accuracy persist in current RAG strategies. A key issue is how to select appropriate methods for user queries of varying complexity dynamically. This study introduces a novel adaptive retrieval-augmented generation framework termed Layered Query Retrieval (LQR). The LQR framework focuses on query complexity classification, retrieval strategies, and relevance analysis, utilizing a custom-built training dataset to train smaller models that aid the large language model (LLM) in efficiently retrieving relevant information. A central technique in LQR is a semantic rule-based approach to distinguish between different levels of multi-hop queries. The process begins by parsing the user’s query for keywords, followed by a keyword-based document retrieval. Subsequently, we employ a natural language inference (NLI) model to assess whether the retrieved document is relevant to the query. We validated our approach on multiple single-hop and multi-hop datasets, demonstrating significant improvements in both accuracy and efficiency compared to existing single-step, multi-step, and adaptive methods. Our method exhibits high accuracy and efficiency, particularly on the HotpotQA dataset, where it outperforms the Adaptive-RAG method by improving accuracy by 9.4% and the F1 score by 16.14%. The proposed approach carefully balances retrieval efficiency with the accuracy of the LLM’s responses.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Information Retrieval and Search Behavior · Expert finding and Q&A systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Agricultural Information Institute, Chinese Academy of Agricultural Sciences, Beijing 100081, China
Key Laboratory of Agricultural Big Data, Ministry of Agriculture and Rural Affairs, Beijing 100081, China
Funding
the National Key Research and Development Program Project · Beijing Smart Agriculture Innovation Consortium Project · Project of Northern Agriculture and Livestock Husbandry Technical Innovation Center, Chinese Academy of Agricultural Sciences · the Chinese Academy of Agricultural Sciences Science and Technology Innovation Project Grant
References (32)
Show source-provided references
1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., and Anadkat, S. (2023). Gpt-4 technical report. arXiv.
2. Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., and Fan, A. (2024). The llama 3 herd of models. arXiv.
3. Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A.M., and Hauth, A. (2023). Gemini: A family of highly capable multimodal models. arXiv.
4. Kandpal, N., Deng, H., Roberts, A., Wallace, E., and Raffel, C. (2023, January 23–29). Large language models struggle to learn long-tail knowledge. Proceedings of the International Conference on Machine Learning, Honolulu, HI, USA.
5. Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., and Wang, H. (2023). Retrieval-augmented generation for large language models: A survey. arXiv.
6. Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., and Chen, Y. (2023). Siren’s song in the AI ocean: A survey on hallucination in large language models. arXiv.
7. Retrieval-augmented generation for knowledge-intensive nlp tasks
8. Li, H., Su, Y., Cai, D., Wang, Y., and Liu, L. (2022). A survey on retrieval-augmented text generation. arXiv.
9. Atlas: Few-shot learning with retrieval augmented language models
10. Tang, Y., and Yang, Y. (2024). Multihop-rag: Benchmarking retrieval-augmented generation for multi-hop queries. arXiv.
11. Lazaridou, A., Gribovskaya, E., Stokowiec, W., and Grigorev, N. (2022). Internet-augmented language models through few-shot prompting for open-domain question answering. arXiv.
12. In-context retrieval-augmented language models10.1162/tacl_a_00605 ↗
13. Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N.A., and Lewis, M. (2022). Measuring and narrowing the compositionality gap in language models. arXiv.10.18653/v1/2023.findings-emnlp.378 ↗
14. Trivedi, H., Balasubramanian, N., Khot, T., and Sabharwal, A. (2022). Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. arXiv.10.18653/v1/2023.acl-long.557 ↗
15. Vu, T., Iyyer, M., Wang, X., Constant, N., Wei, J., Wei, J., Tar, C., Sung, Y.-H., Zhou, D., and Le, Q. (2023). Freshllms: Refreshing large language models with search engine augmentation. arXiv.10.18653/v1/2024.findings-acl.813 ↗
16. Zhao, P., Zhang, H., Yu, Q., Wang, Z., Geng, Y., Fu, F., Yang, L., Zhang, W., and Cui, B. (2024). Retrieval-augmented generation for ai-generated content: A survey. arXiv.
17. Liu, Y., Peng, X., Zhang, X., Liu, W., Yin, J., Cao, J., and Du, T. (2024). RA-ISF: Learning to Answer and Understand from Retrieval Augmentation via Iterative Self-Feedback. arXiv.10.18653/v1/2024.findings-acl.281 ↗
18. Yoran, O., Wolfson, T., Ram, O., and Berant, J. (2023). Making retrieval-augmented language models robust to irrelevant context. arXiv.
19. Zhang, T., Patil, S.G., Jain, N., Shen, S., Zaharia, M., Stoica, I., and Gonzalez, J.E. (2024). Raft: Adapting language model to domain specific rag. arXiv.
20. Shao, Z., Gong, Y., Shen, Y., Huang, M., Duan, N., and Chen, W. (2023). Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy. arXiv.10.18653/v1/2023.findings-emnlp.620 ↗
21. Huang, J., Ping, W., Xu, P., Shoeybi, M., Chang, K.C.-C., and Catanzaro, B. (2023). Raven: In-context learning with retrieval augmented encoder-decoder language models. arXiv.
22. Yu, W., Zhang, H., Pan, X., Ma, K., Wang, H., and Yu, D. (2023). Chain-of-note: Enhancing robustness in retrieval-augmented language models. arXiv.
23. Jeong, S., Baek, J., Cho, S., Hwang, S.J., and Park, J.C. (2024). Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity. arXiv.10.18653/v1/2024.naacl-long.389 ↗
24. Rajpurkar, P. (2016). Squad: 100,000+ questions for machine comprehension of text. arXiv.10.18653/v1/D16-1264 ↗
25. Natural questions: A benchmark for question answering research10.1162/tacl_a_00276 ↗
26. Joshi, M., Choi, E., Weld, D., and Zettlemoyer, L. (August, January 30). TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, BC, Canada.
27. ♫ MuSiQue: Multihop Questions via Single-hop Question Composition10.1162/tacl_a_00475 ↗
28. Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., and Manning, C.D. (November, January 31). HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1259 ↗
29. Ho, X., Nguyen, A.-K.D., Sugawara, S., and Aizawa, A. (2020). Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. arXiv.10.18653/v1/2020.coling-main.580 ↗
30. Beltagy, I., Peters, M.E., and Cohan, A. (2020). Longformer: The long-document transformer. arXiv.
31. Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., and Hajishirzi, H. (2023, January 9–14). When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, Toronto, ON, Canada.10.18653/v1/2023.acl-long.546 ↗
32. Asai, A., Wu, Z., Wang, Y., Sil, A., and Hajishirzi, H. (2023). Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv.