Automated Validating and Fixing of Text-to-SQL Translation with Execution Consistency
Proceedings of the ACM on Management of Data · 17.06.2025 · 3 (3) · 1-28
DOI: 10.1145/3725271 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the ACM on Management of Data, online erschienen am 18.06.2025. Verfasst von Yicun Yang und 7 weiteren Autor:innen. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 66 Literaturangaben. Gefördert von: National Natural Science Foundation of China, National Science Foundation.
Abstract (Originalsprache)
State-of-the-art Text-to-SQL models rely on fine-tuning or few-shot prompting to help LLMs learn from training datasets containing mappings from natural language (NL) queries to SQL statements. Consequently, the quality of the dataset can greatly affect the accuracy of these Text-to-SQL models. Unlike other NL tasks, Text-to-SQL datasets are prone to errors despite extensive manual efforts due to the subtle semantics of SQL. Our study has found a non-negligible (>30%) portion of incorrect NL to SQL mapping cases exists in popular datasets Spider and BIRD. This paper aims to improve the quality of Text-to-SQL training datasets and thereby increase the accuracy of the resulting models. To do so, we propose a necessary correctness condition called execution consistency. For a given database instance, an NL to SQL mapping satisfies execution consistency if the execution result of an NL query matches that of the corresponding SQL. We develop SQLDriller to detect incorrect NL to SQL mappings based on execution consistency in a best-effort manner by crafting database instances that likely result in violations of execution consistency. It generates multiple candidate SQL predictions that differ in their syntax structures. Using a SQL equivalence checker, SQLDriller obtains counterexample database instances that can distinguish non-equivalent candidate SQLs. It then checks the execution consistency of an NL to SQL mapping under this set of counterexamples. The evaluation shows SQLDriller effectively detects and fixes incorrect mappings in the Text-to-SQL dataset, and it improves the model accuracy by up to 13.6%.
Förderung
- National Natural Science Foundation of China
- National Science Foundation
Zitieren
APA
Yang, Y., Wang, Z., Xia, Y., Wei, Z., Ding, H., Piskac, R., Chen, H., & Li, J. (2025). Automated Validating and Fixing of Text-to-SQL Translation with Execution Consistency. Proceedings of the ACM on Management of Data, 3(3), 1-28. https://doi.org/10.1145/3725271
Vancouver
Yang Y, Wang Z, Xia Y, Wei Z, Ding H, Piskac R, et al. Automated Validating and Fixing of Text-to-SQL Translation with Execution Consistency. Proceedings of the ACM on Management of Data. 2025;3(3):1-28. doi:10.1145/3725271
BibTeX
@article{Yang2025,
author = {Yang, Yicun and Wang, Zhaoguo and Xia, Yu and Wei, Zhuoran and Ding, Haoran and Piskac, Ruzica and Chen, Haibo and Li, Jinyang},
title = {Automated Validating and Fixing of Text-to-SQL Translation with Execution Consistency},
journal = {Proceedings of the ACM on Management of Data},
year = {2025},
volume = {3},
number = {3},
pages = {1--28},
doi = {10.1145/3725271}
}
EUVIMED