Validating LLM-Generated SQL Queries through Metamorphic Prompting
Proceedings of the ACM on Software Engineering · 30.06.2026 · 3 (FSE) · 411-434
DOI: 10.1145/3797146 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the ACM on Software Engineering, online erschienen am 30.06.2026. Verfasst von Li Lin und 5 weiteren Autor:innen. Die Arbeit nennt 74 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Natural Science Foundation of China, Fujian Provincial Natural Science Foundation of China.
Abstract (Originalsprache)
Large Language Models (LLMs) can translate natural language (NL) into SQL, enabling non-experts to query databases via conversational interfaces. However, the generated SQL often contains intent-violating hallucinations —queries that are syntactically valid and executable, yet semantically misaligned with the user’s question. These failures are especially risky in real-world settings where users cannot verify the correctness. In this paper, we propose MRSQLGen, a framework for detecting intent-violating hallucinations, built on the metamorphic prompting paradigm. MRSQLGen rewrites the input prompt using task-specific transformation rules derived from a hallucination taxonomy, and validates the generated SQL by checking behavioral consistency across multiple executions. Each transformation is associated with a metamorphic relationship (MR) that defines the expected relation between results; discrepancies are aggregated through a majority-vote strategy to robustly flag hallucinations without ground-truth SQL. We evaluate MRSQLGen on two benchmarks (Spider and Bird) using five representative LLMs, including GPT-4o. Experimental results demonstrate that MRSQLGen consistently outperforms state-of-the-art hallucination detection techniques, achieving higher precision and recall in detecting hallucinated SQL queries.
Förderung
- Natural Science Foundation of China
- Fujian Provincial Natural Science Foundation of China
Zitieren
APA
Lin, L., Zhu, Q., Hong, J., Wang, C., Liu, Y., & Wu, R. (2026). Validating LLM-Generated SQL Queries through Metamorphic Prompting. Proceedings of the ACM on Software Engineering, 3(FSE), 411-434. https://doi.org/10.1145/3797146
Vancouver
Lin L, Zhu Q, Hong J, Wang C, Liu Y, Wu R. Validating LLM-Generated SQL Queries through Metamorphic Prompting. Proceedings of the ACM on Software Engineering. 2026;3(FSE):411-434. doi:10.1145/3797146
BibTeX
@article{Lin2026,
author = {Lin, Li and Zhu, Qinglin and Hong, Jintai and Wang, Chong and Liu, Yang and Wu, Rongxin},
title = {Validating LLM-Generated SQL Queries through Metamorphic Prompting},
journal = {Proceedings of the ACM on Software Engineering},
year = {2026},
volume = {3},
number = {FSE},
pages = {411--434},
doi = {10.1145/3797146}
}
EUVIMED