Schema Retrieval with Embeddings and Vector Stores Using Retrieval-Augmented Generation and LLM-Based SQL Query Generation
Source abstract
Abstract
In today’s world, where the volume and variety of data are increasing at an extraordinary rate, extracting meaningful insights from data is of critical importance; however, the complexity of standard database query languages makes it difficult for users without technical expertise to access information. This study proposes an innovative Retrieval-Augmented Generation (RAG) architecture that analyzes natural language queries, identifies related database schemas, and automatically converts them to SQL. Unlike fixed schema selection (fixed-k) methods, a unique hierarchical clustering mechanism is introduced to dynamically determine the number of relevant schemas, minimizing noise. Furthermore, the architecture incorporates an iterative repair mechanism, data enrichment with sample rows, and a hybrid query strategy (Turkish + English) to overcome cross-lingual barriers. Performance evaluations on 15 databases demonstrate that the proposed method improved the schema retrieval F1 score from 0.79 to 0.88. In the SQL generation phase, the execution accuracy (EX) of the GPT-4o model increased from 0.70 to 0.78 with the proposed optimizations, representing an approximate 11% improvement relative to the baseline configuration without requiring fine-tuning.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Data Quality and Management · Advanced Database Systems and Queries · Web Data Mining and Analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Engineering, Bursa Uludağ University, 16059 Bursa, Turkey
Funding
Bursa Uludağ University Science and Technology Centre
References (32)
Show source-provided references
1. Minaee, S., Mikolov, T., Nikzad, N., Chenaghlu, M., Socher, R., Amatriain, X., and Gao, J. (2024). Large Language Models: A Survey. arXiv, Available online: http://arxiv.org/abs/2402.06196.
2. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention Is All You Need. arXiv, Available online: http://arxiv.org/abs/1706.03762.
3. Naveed, H., Khan, A.U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., and Mian, A. (2023). A Comprehensive Overview of Large Language Models. arXiv, Available online: http://arxiv.org/abs/2307.06435.
4. Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., and Wang, H. (2023). Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv, Available online: http://arxiv.org/abs/2312.10997.
5. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., and Rocktäschel, T. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv, Available online: http://arxiv.org/abs/2005.11401.
6. Wang, M., Shafran, I., Soltau, H., Han, W., Cao, Y., Yu, D., and El Shafey, L. (2024, January 14–19). Retrieval Augmented End-to-End Spoken Dialog Models. Proceedings of the 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul, Republic of Korea.10.1109/ICASSP48485.2024.10447448 ↗
7. Ma, L., Zhang, R., Han, Y., Yu, S., Wang, Z., Ning, Z., Zhang, J., Xu, P., Li, P., and Ju, W. (2023). A Comprehensive Survey on Vector Database: Storage and Retrieval Technique, Challenge. arXiv, Available online: http://arxiv.org/abs/2310.11703.
8. Hong, Z., Yuan, Z., Zhang, Q., Chen, H., Dong, J., Huang, F., and Huang, X. (2024). Next-Generation Database Interfaces: A Survey of LLM-Based Text-to-SQL. arXiv, Available online: http://arxiv.org/abs/2406.08426.
9. Zhu, X., Li, Q., Cui, L., and Liu, Y. (2024). Large Language Model Enhanced Text-to-SQL Generation: A Survey. arXiv, Available online: http://arxiv.org/abs/2410.06011.
10. A Survey on Employing Large Language Models for Text-to-SQL Tasks
11. Kanburoglu, A.B., and Tek, F.B. (2023). TUR2SQL: A Cross-Domain Turkish Dataset for Text-to-SQL. UBMK 2023—Proceedings: 8th International Conference on Computer Science and Engineering, Burdur, Turkiye, 13–15 September 2023, Institute of Electrical and Electronics Engineers Inc.10.1109/UBMK59864.2023.10286686 ↗
12. Text-to-SQL: A methodical review of challenges and models10.55730/1300-0632.4077 ↗
13. Wang, C., Tatwawadi, K., Brockschmidt, M., Huang, P.S., Mao, Y., Polozov, O., and Singh, R. (2018). Robust Text-to-SQL Generation with Execution-Guided Decoding. arXiv, Available online: http://arxiv.org/abs/1807.03100.
14. Kumar, R., Amar, A., Dibbu, R., Harsola, S.H., Subrahmaniam, V., and Modi, A. (2024). BookSQL: A Large Scale Text-to-SQL Dataset for Accounting Domain. Human Language Technologies (Volume 1: Long Papers), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics, Mexico City, Mexico, 16–21 June 2024, Association for Computational Linguistics.10.18653/v1/2024.naacl-long.28 ↗
15. Rushdy, M.R.A., and Thayasivam, U. (2023). Application of Noise Filter Mechanism for T5-Based Text-To-SQL Generation. MERCon, Proceedings of the Moratuwa Engineering Research Conference, Moratuwa, Sri Lanka, 9–11 November 2023, Institute of Electrical and Electronics Engineers Inc.
16. Demirkiran, F., Coşkun, A.K., Kömeçoğlu, Y., Kömeçoğlu, B.B., and Güven, R. (2024, January 26–28). Enhancing Text-to-SQL Conversion in Turkish: An Analysis of LLMs with Schema Context. Proceedings of the 2024 9th International Conference on Computer Science and Engineering (UBMK), Antalya, Turkiye.10.1109/UBMK63289.2024.10773403 ↗
17. DBTagger: Multi-task learning for keyword mapping in NLIDBs using bi-directional recurrent neural networks10.14778/3446095.3446103 ↗
18. Zhang, K., Lin, X., Wang, Y., Zhang, X., Sun, F., Cen, J., Tan, H., Jiang, X., and Shen, H. (2023). ReFSQL: A Retrieval-Augmentation Framework for Text-to-SQL Generation. Findings of the Association for Computational Linguistics: EMNLP 2023, Association for Computational Linguistics.10.18653/v1/2023.findings-emnlp.48 ↗
19. Wang, B., Ren, C., Yang, J., Liang, X., Bai, J., Chai, L., Yan, Z., Zhang, Q.-W., Yin, D., and Sun, X. (2025, January 19–24). MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQL. Proceedings of the 31st International Conference on Computational Linguistics (COLING 2025), Abu Dhabi, United Arab Emirates. Available online: https://github.com/wbbeyourself/MAC-SQL.
20. Pourreza, M., and Rafiei, D. (2024). DTS-SQL: Decomposed Text-to-SQL with Small Large Language Models. arXiv, Available online: http://arxiv.org/abs/2402.01117.10.18653/v1/2024.findings-emnlp.481 ↗
21. Vu, T., Balasubramaniam, T., Nayak, R., and Keretna, S. (2024). Improved Productivity with AI Models for SQL Tasks: A Case Study. Computer Science & Information Technology (CS & IT), Academy and Industry Research Collaboration Center (AIRCC).10.5121/csit.2024.142402 ↗
22. Li, J., Hui, B., Qu, G., Yang, J., Li, B., Li, B., Wang, B., Qin, B., Geng, R., and Huo, N. (2023). Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs. arXiv, Available online: http://arxiv.org/abs/2305.03111.
23. Gao, D., Wang, H., Li, Y., Sun, X., Qian, Y., Ding, B., and Zhou, J. (2023). Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation. arXiv, Available online: http://arxiv.org/abs/2308.15363.
24. (2024, October 01). Spider_Bird: Combined Spider and BIRD Dataset for Text-to-SQL. Available online: https://huggingface.co/datasets/starrysky9959/spider_bird.
25. Almeida, F., and Xexéo, G. (2019). Word Embeddings: A Survey. arXiv, Available online: http://arxiv.org/abs/1901.09069.
26. Mikolov, T., Sutskever, I., Chen, K., Corrado, G., and Dean, J. (2013). Distributed Representations of Words and Phrases and their Compositionality. arXiv, Available online: http://arxiv.org/abs/1310.4546.
27. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv, Available online: http://arxiv.org/abs/1301.3781.
28. Yang, H., Zhang, M., and Wei, D. (2023). SRAG: Speech Retrieval Augmented Generation for Spoken Language Understanding. ISRITI 2023, Proceedings of the 6th International Seminar on Research of Information Technology and Intelligent Systems, Batam, Indonesia, 11 December 2023, Institute of Electrical and Electronics Engineers Inc.10.1109/ISRITI60336.2023.10467285 ↗
29. Lenssen, L., Schubert, E., Krivošija, A., Schubert, E., Lang, A., and Hess, S. (2022). 5 Cluster Analysis. Fundamentals, De Gruyter.10.1515/9783110785944-005 ↗
30. Dospinescu, O., and Buraga, S. (2025). Integrated ERP Systems—Determinant Factors for Their Adoption in Romanian Organizations. Systems, 13.10.3390/systems13080667 ↗
31. Big Data in Leadership Studies: Automated Machine Learning Model to Predict Preferred Leader Behavior Across Cultures10.3390/businesses4040039 ↗
32. Selection of software tools for automation of business planning processes10.30977/ETK.2225-2304.2023.41.78 ↗