TPCx-AI under the Microscope: A Benchmarking Debt Analysis
Proceedings of the VLDB Endowment · 01.02.2026 · 19 (6) · 1305-1318
DOI: 10.14778/3797919.3797936 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the VLDB Endowment, online erschienen am 08.05.2026. Verfasst von Ilin Tolovski und 3 weiteren Autor:innen. Die Arbeit nennt 41 Literaturangaben.
Abstract (Originalsprache)
TPCx-AI is an industry standard benchmark for evaluating the end-to-end performance of machine learning systems and the underlying hardware configurations. In the database community, individual parts of the dataset and the workloads are used to evaluate preprocessing methods and systems for fast inference. In both of these cases, the datasets and workloads are used based on the characteristics defined in the specification. Upon analysis of TPCx-AI's dataset and use cases, we observe that the official implementation of TPCx-AI's kit diverges from the specification, does not evaluate the capabilities of the system under test, and impacts the overall performance in a benchmark run. In this paper, we investigate the benchmarking debt accumulated in the TPCx-AI dataset and the workloads. We identify properties that impact the benchmark's performance, including runtime and quality of use cases, the defined metrics and their thresholds, workload discrepancies, and data errors. Our analysis shows that all use cases and datasets contain benchmarking debts, impacting the training and serving runtimes by up to 350× and 800×, respectively. By addressing these debts, we observe an end-to-end throughput increase of up to 3.8× over the default TPCx-AI implementation.
Zitieren
APA
Tolovski, I., Hildebrandt, P., Daudjee, K., & Rabl, T. (2026). TPCx-AI under the Microscope: A Benchmarking Debt Analysis. Proceedings of the VLDB Endowment, 19(6), 1305-1318. https://doi.org/10.14778/3797919.3797936
Vancouver
Tolovski I, Hildebrandt P, Daudjee K, Rabl T. TPCx-AI under the Microscope: A Benchmarking Debt Analysis. Proceedings of the VLDB Endowment. 2026;19(6):1305-1318. doi:10.14778/3797919.3797936
BibTeX
@article{Tolovski2026,
author = {Tolovski, Ilin and Hildebrandt, Philipp and Daudjee, Khuzaima and Rabl, Tilmann},
title = {TPCx-AI under the Microscope: A Benchmarking Debt Analysis},
journal = {Proceedings of the VLDB Endowment},
year = {2026},
volume = {19},
number = {6},
pages = {1305--1318},
doi = {10.14778/3797919.3797936}
}
EUVIMED