Self-Correcting Text-to-SQL Agent with Error Feedback: A Reproducible Closed-Loop Evaluation on Compact Executable SQLite Benchmarks
Journal of Advanced Computing Systems · 21.11.2024 · 4 (11) · 86-104
DOI: 10.69987/jacs.2024.41107 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Advanced Computing Systems, online erschienen am 21.11.2024. Verfasst von Siyu Chen und 2 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Text-to-SQL systems translate natural-language questions into executable database queries, but first-pass predictions often fail because of syntax errors, schema-linking mistakes, missing joins, wrong aggregation, value typos, and semantically valid but wrong column choices. This paper presents EFRA, an error-feedback repair agent that closes the loop between SQL generation, SQLite execution, and deterministic correction. EFRA receives a question, schema, and first-pass SQL candidate; executes the candidate; parses SQLite errors or empty-like results; repairs syntax, table names, column names, join paths, aggregators, predicates, grouping, limits, and values; and repeats the process under a fixed feedback-turn budget. To avoid illustrative results, the study conducts a full empirical evaluation on two compact executable datasets included with the artifact: ExecSpiderLite, a 240-example cross-domain multi-table benchmark, and ExecWikiSQLLite, a 180-example single-table benchmark. The evaluation executes every gold and predicted query over 14 SQLite databases and logs exact match, validity, execution accuracy, number of executions, and latency. EFRA reaches 90.7% execution accuracy and 100.0% validity over all 420 examples, compared with 13.1% execution accuracy for the first-pass generator, 19.8% for a syntax-only guard, and 55.0% for execution-only repair. Ablations show that semantic validators and value repair are both necessary: removing semantic validators reduces execution accuracy to 55.0%, and removing value repair reduces it to 79.3%. The results establish a concise, reproducible baseline for agentic data access in which execution feedback is treated as a first-class signal rather than a post-hoc check.
Zitieren
APA
Chen, S., Su, W., & Ma, J. (2024). Self-Correcting Text-to-SQL Agent with Error Feedback: A Reproducible Closed-Loop Evaluation on Compact Executable SQLite Benchmarks. Journal of Advanced Computing Systems, 4(11), 86-104. https://doi.org/10.69987/jacs.2024.41107
Vancouver
Chen S, Su W, Ma J. Self-Correcting Text-to-SQL Agent with Error Feedback: A Reproducible Closed-Loop Evaluation on Compact Executable SQLite Benchmarks. Journal of Advanced Computing Systems. 2024;4(11):86-104. doi:10.69987/jacs.2024.41107
BibTeX
@article{Chen2024,
author = {Chen, Siyu and Su, Wenhao and Ma, Jacob},
title = {Self-Correcting Text-to-SQL Agent with Error Feedback: A Reproducible Closed-Loop Evaluation on Compact Executable SQLite Benchmarks},
journal = {Journal of Advanced Computing Systems},
year = {2024},
volume = {4},
number = {11},
pages = {86--104},
doi = {10.69987/jacs.2024.41107}
}
EUVIMED