Refining Zero-Shot Text-to-SQL Benchmarks via Prompt Strategies with Large Language Models
Source abstract
Abstract
Text-to-SQL leverages large language models (LLMs) for natural language database queries, yet existing benchmarks like BIRD (12,751 question–SQL pairs, 95 databases) suffer from inconsistencies—e.g., 30% of queries misalign with SQL outputs—and ambiguities that impair LLM evaluation. This study refines such datasets by distilling logically sound question–SQL pairs and enhancing table schemas, yielding a benchmark of 146 high-complexity tasks across 11 domains. We assess GPT-4o, GPT-4o-Mini, Qwen-2.5-Instruct, llama 370b, DPSK-v3 and O1-Preview in zero-shot scenarios, achieving average accuracies of 51.23%, 41.65%, 44.25%, 47.80%, and 49.10% and a peak of 78.08% (O1-Preview), respectively. Prompt-based strategies improve performance by up to 4.78%, addressing issues like poor domain adaptability and inconsistent training data interpretation. Error-annotated datasets further reveal LLM limitations. This refined benchmark ensures robust evaluation of logical reasoning, supporting reliable NLP-driven database systems.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Web Data Mining and Analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computation, Information and Technology, Technical University of Munich, 80333 Munich, Germany
State Key Laboratory of Ocean Sensing & Ocean College, Zhejiang University, Zhoushan 316021, China
Kavli Institute for Astrophysics and Space Research Center, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
Funding
Ministry of Science and Technology of the People’s Republic of China · National Natural Science Foundation of China
References (37)
Show source-provided references
1. Zhong, V., Xiong, C., and Socher, R. (2017). Seq2SQL: Generating structured queries from natural language using reinforcement learning. arXiv.
2. Xu, X., Liu, C., and Yu, D. (2017). SQLNet: Generating structured queries from natural language without reinforcement learning. arXiv.
3. Yu, T., Yasunaga, M., Yang, K., Zhang, R., Wang, D., Li, Z., and Radev, D. (2018). SyntaxSQLNet: Syntax tree networks for complex and cross-domaintext-to-SQL task. arXiv.10.18653/v1/D18-1193 ↗
4. Gao, D., Wang, H., Li, Y., Sun, X., Qian, Y., Ding, B., and Zhou, J. (2023). Text-to-SQL empowered by large language models: A benchmark evaluation. arXiv.10.14778/3641204.3641221 ↗
5. Sun, R., Arik, S.Ö., Muzio, A., Miculicich, L., Gundabathula, S., Yin, P., Dai, H., Nakhost, H., Sinha, R., and Wang, Z. (2023). SQL-PaLM: Improved large language model adaptation for text-to-SQL. arXiv.
6. Hong, Z., Yuan, Z., Zhang, Q., Chen, H., Dong, J., Huang, F., and Huang, X. (2024). Next-generation database interfaces: A survey of llm-based text-to-SQL. arXiv.
7. Shi, L., Tang, Z., Zhang, N., Zhang, X., and Yang, Z. (2023). A survey on employing large language models for text-to-SQL tasks. arXiv.
8. Language models are few-shot learners
9. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., and Anadkat, S. (2023). Gpt-4 technical report. arXiv.
10. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., and Azhar, F. (2023). Llama: Open and efficient foundation language models. arXiv.
11. Liu, A., Hu, X., Wen, L., and Yu, P.S. (2023). A comprehensive evaluation of chatgpt’s zero-shot text-to-SQL capability. arXiv.
12. Li, J., Hui, B., Qu, G., Yang, J., Li, B., Li, B., Wang, B., Qin, B., Geng, R., and Huo, N. (2023). Can llm already serve as a database interface? a big bench for large-scale database grounded textto-SQLs. arXiv.
13. Dong, X., Zhang, C., Ge, Y., Mao, Y., Gao, Y., Lin, J., and Lou, D. (2023). C3: Zero-shot text-to-SQL with chatgpt. arXiv.
14. Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., and Roman, S. (2018). Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. arXiv.10.18653/v1/D18-1425 ↗
15. Lei, F., Chen, J., Ye, Y., Cao, R., Shin, D., Su, H., Suo, Z., Gao, H., Hu, W., and Yin, P. (2024). Spider 2.0: Evaluating language models on real-world enterprise text-to-SQL workflows. arXiv.
16. Yu, T., Zhang, R., Yasunaga, M., Tan, Y.C., Lin, X.V., Li, S., Er, H., Li, I., Pang, B., and Chen, T. (2019). Sparc: Crossdomain semantic parsing in context. arXiv.10.18653/v1/P19-1443 ↗
17. Yu, T., Zhang, R., Er, H.Y., Li, S., Xue, E., Pang, B., Lin, X.V., Tan, Y.C., Shi, T., and Li, Z. (2019). CoSQL: A conversational text-to-SQL challenge towards cross-domain natural language interfaces to databases. arXiv.10.18653/v1/D19-1204 ↗
18. Dahl, D.A., Bates, M., Brown, M., Fisher, W., Hunicke-Smith, K., Pallett, D., Pao, C., Rudnicky, A., and Shriberg, E. (1994, January 8–11). Expanding the scope of the ATIS task: The ATIS-3 corpus. Proceedings of the Human Language Technology: Proceedings of a Workshop, Plainsboro, NJ, USA.10.3115/1075812.1075823 ↗
19. Kelkar, A., Relan, R., Bhardwaj, V., Vaichal, S., Khatri, C., and Relan, P. (2020). Bertrand-dr: Improving text-to-SQL using a discriminative re-ranker. arXiv.
20. Scholak, T., Schucher, N., and Bahdanau, D. (2021). PICARD: Parsing incrementally for constrained auto-regressive decoding from language models. arXiv.10.18653/v1/2021.emnlp-main.779 ↗
21. Anil, R., Dai, A.M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., and Chen, Z. (2023). PaLM 2 technical report. arXiv.
22. Rajkumar, N., Li, R., and Bahdanau, D. (2022). Evaluating the text-to-SQL capabilities of large language models. arXiv.
23. Wang, B., Ren, C., Yang, J., Liang, X., Bai, J., Chai, L., Yan, Z., Zhang, Q.W., Yin, D., and Sun, X. (2024). Mac-SQL: A multi-agent collaborative framework for text-to-SQL. arXiv.
24. Cen, J., Liu, J., Li, Z., and Wang, J. (2024). SQLfixagent: Towards semantic-accurate text-to-SQL parsing via consistency-enhanced multi-agent collaboration. arXiv.10.1609/aaai.v39i1.31979 ↗
25. Tai, C.Y., Chen, Z., Zhang, T., Deng, X., and Sun, H. (2023). Exploring chain of thought style prompting for text-to-SQL. Empirical Methods in Natural Language Processing (EMNLP). arXiv.10.18653/v1/2023.emnlp-main.327 ↗
26. Codes: Towards building opensource language models for text-to-SQL
27. Li, C., Shao, Y., Li, Y., and Liu, Z. (2024). Sea-SQL: Semantic-enhanced text-to-SQL with adaptive refinement. arXiv.
28. Sun, Y., Guo, Z., Yu, H., Liu, C., Li, X., Wang, B., Yu, X., and Zhao, T. (2024). Qda-SQL: Questions enhanced dialogue augmentation for multi-turn text-to-SQL. arXiv.
29. Xie, X., Xu, G., Zhao, L., and Guo, R. (2025). Opensearch-SQL: Enhancing text-to-SQL with dynamic few-shot and consistency alignment. arXiv.
30. Lee, D., Park, C., Kim, J., and Park, H. (2024). Mcs-SQL: Leveraging multiple prompts and multiple-choice selection for text-to-SQL generation. arXiv.
31. Xie, W., Wu, G., and Zhou, B. (2024). Mag-SQL: Multi-agent generative approach with soft schema linking and iterative sub-SQL refinement for text-to-SQL. arXiv.
32. Few-shot text-to-SQL translation using structure and content prompt learning
33. Zhu, X., Li, Q., Cui, L., and Liu, Y. (2024). Large language model enhanced textto-SQL generation: A survey. arXiv.
34. Xia, H., Jiang, F., Deng, N., Wang, C., Zhao, G., Mihalcea, R., and Zhang, Y. (2024). SQL-craft: Text-to-SQL through interactive refinement and enhanced reasoning. arXiv.
35. Nan, L., Zhao, Y., Zou, W., Ri, N., Tae, J., Zhang, E., Cohan, A., and Radev, D. (2023). Enhancing text-toSQL capabilities of large language models: A study on prompt design strategies. Findings of Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics.10.18653/v1/2023.findings-emnlp.996 ↗
36. Wu, Z., Zhu, F., Shang, X., Zhang, Y., and Zhou, P. (2024). Cooperative SQL generation for segmented databases by using multi-functional llm agents. arXiv.
37. Chang, S., and Fosler-Lussier, E. (2023). Selective demonstrations for crossdomain text-to-SQL. arXiv.10.18653/v1/2023.findings-emnlp.944 ↗