EUVIMEDEUVIMED entdecken
Crossref

Automated Validating and Fixing of Text-to-SQL Translation with Execution Consistency

Yicun Yang, Zhaoguo Wang, Yu Xia, Zhuoran Wei, Haoran Ding, Ruzica Piskac, Haibo Chen, Jinyang Li

Proceedings of the ACM on Management of Data · 17.06.2025 · 3 (3) · 1-28

DOI: 10.1145/3725271 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the ACM on Management of Data, online erschienen am 18.06.2025. Verfasst von Yicun Yang und 7 weiteren Autor:innen. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 66 Literaturangaben. Gefördert von: National Natural Science Foundation of China, National Science Foundation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

State-of-the-art Text-to-SQL models rely on fine-tuning or few-shot prompting to help LLMs learn from training datasets containing mappings from natural language (NL) queries to SQL statements. Consequently, the quality of the dataset can greatly affect the accuracy of these Text-to-SQL models. Unlike other NL tasks, Text-to-SQL datasets are prone to errors despite extensive manual efforts due to the subtle semantics of SQL. Our study has found a non-negligible (>30%) portion of incorrect NL to SQL mapping cases exists in popular datasets Spider and BIRD. This paper aims to improve the quality of Text-to-SQL training datasets and thereby increase the accuracy of the resulting models. To do so, we propose a necessary correctness condition called execution consistency. For a given database instance, an NL to SQL mapping satisfies execution consistency if the execution result of an NL query matches that of the corresponding SQL. We develop SQLDriller to detect incorrect NL to SQL mappings based on execution consistency in a best-effort manner by crafting database instances that likely result in violations of execution consistency. It generates multiple candidate SQL predictions that differ in their syntax structures. Using a SQL equivalence checker, SQLDriller obtains counterexample database instances that can distinguish non-equivalent candidate SQLs. It then checks the execution consistency of an NL to SQL mapping under this set of counterexamples. The evaluation shows SQLDriller effectively detects and fixes incorrect mappings in the Text-to-SQL dataset, and it improves the model accuracy by up to 13.6%.

Förderung

  • National Natural Science Foundation of China
  • National Science Foundation

Zitieren

APA

Yang, Y., Wang, Z., Xia, Y., Wei, Z., Ding, H., Piskac, R., Chen, H., & Li, J. (2025). Automated Validating and Fixing of Text-to-SQL Translation with Execution Consistency. Proceedings of the ACM on Management of Data, 3(3), 1-28. https://doi.org/10.1145/3725271

Vancouver

Yang Y, Wang Z, Xia Y, Wei Z, Ding H, Piskac R, et al. Automated Validating and Fixing of Text-to-SQL Translation with Execution Consistency. Proceedings of the ACM on Management of Data. 2025;3(3):1-28. doi:10.1145/3725271

BibTeX
@article{Yang2025,
  author = {Yang, Yicun and Wang, Zhaoguo and Xia, Yu and Wei, Zhuoran and Ding, Haoran and Piskac, Ruzica and Chen, Haibo and Li, Jinyang},
  title = {Automated Validating and Fixing of Text-to-SQL Translation with Execution Consistency},
  journal = {Proceedings of the ACM on Management of Data},
  year = {2025},
  volume = {3},
  number = {3},
  pages = {1--28},
  doi = {10.1145/3725271}
}
Kostenlos starten →