EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Self-Correcting Text-to-SQL Agent with Error Feedback: A Reproducible Closed-Loop Evaluation on Compact Executable SQLite Benchmarks

Siyu Chen, Wenhao Su, Jacob Ma

Journal of Advanced Computing Systems · 21.11.2024 · 4 (11) · 86-104

DOI: 10.69987/jacs.2024.41107 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Advanced Computing Systems, online erschienen am 21.11.2024. Verfasst von Siyu Chen und 2 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Text-to-SQL systems translate natural-language questions into executable database queries, but first-pass predictions often fail because of syntax errors, schema-linking mistakes, missing joins, wrong aggregation, value typos, and semantically valid but wrong column choices. This paper presents EFRA, an error-feedback repair agent that closes the loop between SQL generation, SQLite execution, and deterministic correction. EFRA receives a question, schema, and first-pass SQL candidate; executes the candidate; parses SQLite errors or empty-like results; repairs syntax, table names, column names, join paths, aggregators, predicates, grouping, limits, and values; and repeats the process under a fixed feedback-turn budget. To avoid illustrative results, the study conducts a full empirical evaluation on two compact executable datasets included with the artifact: ExecSpiderLite, a 240-example cross-domain multi-table benchmark, and ExecWikiSQLLite, a 180-example single-table benchmark. The evaluation executes every gold and predicted query over 14 SQLite databases and logs exact match, validity, execution accuracy, number of executions, and latency. EFRA reaches 90.7% execution accuracy and 100.0% validity over all 420 examples, compared with 13.1% execution accuracy for the first-pass generator, 19.8% for a syntax-only guard, and 55.0% for execution-only repair. Ablations show that semantic validators and value repair are both necessary: removing semantic validators reduces execution accuracy to 55.0%, and removing value repair reduces it to 79.3%. The results establish a concise, reproducible baseline for agentic data access in which execution feedback is treated as a first-class signal rather than a post-hoc check.

Zitieren

APA

Chen, S., Su, W., & Ma, J. (2024). Self-Correcting Text-to-SQL Agent with Error Feedback: A Reproducible Closed-Loop Evaluation on Compact Executable SQLite Benchmarks. Journal of Advanced Computing Systems, 4(11), 86-104. https://doi.org/10.69987/jacs.2024.41107

Vancouver

Chen S, Su W, Ma J. Self-Correcting Text-to-SQL Agent with Error Feedback: A Reproducible Closed-Loop Evaluation on Compact Executable SQLite Benchmarks. Journal of Advanced Computing Systems. 2024;4(11):86-104. doi:10.69987/jacs.2024.41107

BibTeX
@article{Chen2024,
  author = {Chen, Siyu and Su, Wenhao and Ma, Jacob},
  title = {Self-Correcting Text-to-SQL Agent with Error Feedback: A Reproducible Closed-Loop Evaluation on Compact Executable SQLite Benchmarks},
  journal = {Journal of Advanced Computing Systems},
  year = {2024},
  volume = {4},
  number = {11},
  pages = {86--104},
  doi = {10.69987/jacs.2024.41107}
}
Kostenlos starten →