Refining Zero-Shot Text-to-SQL Benchmarks via Prompt Strategies with Large Language Models
Applied Sciences · 09.05.2025 · 15 (10) · 5306
DOI: 10.3390/app15105306 ↗Kurz gefasst
Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 09.05.2025. Verfasst von Ruikang Zhou und 1 weiteren Autor:innen. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 37 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Ministry of Science and Technology of the People’s Republic of China, National Natural Science Foundation of China.
Abstract (Originalsprache)
Text-to-SQL leverages large language models (LLMs) for natural language database queries, yet existing benchmarks like BIRD (12,751 question–SQL pairs, 95 databases) suffer from inconsistencies—e.g., 30% of queries misalign with SQL outputs—and ambiguities that impair LLM evaluation. This study refines such datasets by distilling logically sound question–SQL pairs and enhancing table schemas, yielding a benchmark of 146 high-complexity tasks across 11 domains. We assess GPT-4o, GPT-4o-Mini, Qwen-2.5-Instruct, llama 370b, DPSK-v3 and O1-Preview in zero-shot scenarios, achieving average accuracies of 51.23%, 41.65%, 44.25%, 47.80%, and 49.10% and a peak of 78.08% (O1-Preview), respectively. Prompt-based strategies improve performance by up to 4.78%, addressing issues like poor domain adaptability and inconsistent training data interpretation. Error-annotated datasets further reveal LLM limitations. This refined benchmark ensures robust evaluation of logical reasoning, supporting reliable NLP-driven database systems.
Förderung
- Ministry of Science and Technology of the People’s Republic of China
- National Natural Science Foundation of China
Zitieren
APA
Zhou, R., & Zhang, F. (2025). Refining Zero-Shot Text-to-SQL Benchmarks via Prompt Strategies with Large Language Models. Applied Sciences, 15(10), 5306. https://doi.org/10.3390/app15105306
Vancouver
Zhou R, Zhang F. Refining Zero-Shot Text-to-SQL Benchmarks via Prompt Strategies with Large Language Models. Applied Sciences. 2025;15(10):5306. doi:10.3390/app15105306
BibTeX
@article{Zhou2025,
author = {Zhou, Ruikang and Zhang, Fan},
title = {Refining Zero-Shot Text-to-SQL Benchmarks via Prompt Strategies with Large Language Models},
journal = {Applied Sciences},
year = {2025},
volume = {15},
number = {10},
pages = {5306},
doi = {10.3390/app15105306}
}
EUVIMED