SQLStorm: Taking Database Benchmarking into the LLM Era
Proceedings of the VLDB Endowment · 2025-07 · 18 (11) · 4144-4157
DOI: 10.14778/3749646.3749683 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the VLDB Endowment, online erschienen am 04.09.2025. Verfasst von Tobias Schmidt und 3 weiteren Autor:innen. Die Arbeit wurde bisher 22-mal zitiert (Crossref) und nennt 70 Literaturangaben.
Abstract (Originalsprache)
In this paper, we introduce a new methodology for constructing database benchmarks using Large Language Models (LLMs), as well as SQLStorm v1.0, a concrete benchmark on a real-world dataset of three sizes (1 GB, 12 GB, 220 GB) consisting of over 18 K queries. This methodology of using AI to generate query workloads breaks new ground, not only in its ability to cheaply ($15) generate huge volumes (22 MB) of realistic queries but especially because it greatly expands the amount of SQL functionality and query constructions that is covered, compared to human-written SQL benchmarks such as TPC-H, TPC-DS, and JOB. The use cases of SQLStorm that we think will advance data systems most are: (i) improving SQL compatibility between systems, (ii) increasing system quality by identifying crashes/errors and fixing those, (iii) improving cardinality estimators and query optimizers, by identifying trends and opportunities (queries where other systems do much better), as well as (iv) overall system performance, both in terms of speed and robustness.
Zitieren
APA
Schmidt, T., Leis, V., Boncz, P., & Neumann, T. (2025). SQLStorm: Taking Database Benchmarking into the LLM Era. Proceedings of the VLDB Endowment, 18(11), 4144-4157. https://doi.org/10.14778/3749646.3749683
Vancouver
Schmidt T, Leis V, Boncz P, Neumann T. SQLStorm: Taking Database Benchmarking into the LLM Era. Proceedings of the VLDB Endowment. 2025;18(11):4144-4157. doi:10.14778/3749646.3749683
BibTeX
@article{Schmidt2025,
author = {Schmidt, Tobias and Leis, Viktor and Boncz, Peter and Neumann, Thomas},
title = {SQLStorm: Taking Database Benchmarking into the LLM Era},
journal = {Proceedings of the VLDB Endowment},
year = {2025},
volume = {18},
number = {11},
pages = {4144--4157},
doi = {10.14778/3749646.3749683}
}
EUVIMED