EUVIMEDEUVIMED entdecken
Crossref

TPCx-AI under the Microscope: A Benchmarking Debt Analysis

Ilin Tolovski, Philipp Hildebrandt, Khuzaima Daudjee, Tilmann Rabl

Proceedings of the VLDB Endowment · 01.02.2026 · 19 (6) · 1305-1318

DOI: 10.14778/3797919.3797936 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the VLDB Endowment, online erschienen am 08.05.2026. Verfasst von Ilin Tolovski und 3 weiteren Autor:innen. Die Arbeit nennt 41 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

TPCx-AI is an industry standard benchmark for evaluating the end-to-end performance of machine learning systems and the underlying hardware configurations. In the database community, individual parts of the dataset and the workloads are used to evaluate preprocessing methods and systems for fast inference. In both of these cases, the datasets and workloads are used based on the characteristics defined in the specification. Upon analysis of TPCx-AI's dataset and use cases, we observe that the official implementation of TPCx-AI's kit diverges from the specification, does not evaluate the capabilities of the system under test, and impacts the overall performance in a benchmark run. In this paper, we investigate the benchmarking debt accumulated in the TPCx-AI dataset and the workloads. We identify properties that impact the benchmark's performance, including runtime and quality of use cases, the defined metrics and their thresholds, workload discrepancies, and data errors. Our analysis shows that all use cases and datasets contain benchmarking debts, impacting the training and serving runtimes by up to 350× and 800×, respectively. By addressing these debts, we observe an end-to-end throughput increase of up to 3.8× over the default TPCx-AI implementation.

Zitieren

APA

Tolovski, I., Hildebrandt, P., Daudjee, K., & Rabl, T. (2026). TPCx-AI under the Microscope: A Benchmarking Debt Analysis. Proceedings of the VLDB Endowment, 19(6), 1305-1318. https://doi.org/10.14778/3797919.3797936

Vancouver

Tolovski I, Hildebrandt P, Daudjee K, Rabl T. TPCx-AI under the Microscope: A Benchmarking Debt Analysis. Proceedings of the VLDB Endowment. 2026;19(6):1305-1318. doi:10.14778/3797919.3797936

BibTeX
@article{Tolovski2026,
  author = {Tolovski, Ilin and Hildebrandt, Philipp and Daudjee, Khuzaima and Rabl, Tilmann},
  title = {TPCx-AI under the Microscope: A Benchmarking Debt Analysis},
  journal = {Proceedings of the VLDB Endowment},
  year = {2026},
  volume = {19},
  number = {6},
  pages = {1305--1318},
  doi = {10.14778/3797919.3797936}
}
Kostenlos starten →