EUVIMEDEUVIMED entdecken
Crossref

SemBench: A Benchmark for Semantic Query Processing Engines

Jiale Lao, Andreas Zimmerer, Olga Ovcharenko, Tianji Cong, Matthew Russo, Gerardo Vitagliano, Michael Cochez, Fatma Özcan, Gautam Gupta, Thibaud Hottelier, H. V. Jagadish, Kris Kissel und 3 weitere

Proceedings of the VLDB Endowment · 01.04.2026 · 19 (8) · 1754-1767

DOI: 10.14778/3811243.3811249 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the VLDB Endowment, online erschienen am 15.07.2026. Verfasst von Jiale Lao und 14 weiteren Autor:innen. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 57 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We present a benchmark targeting a novel class of systems: semantic query processing engines. Those systems rely inherently on generative and reasoning capabilities of state-of-the-art large language models (LLMs). They extend SQL with semantic operators, configured by natural language instructions, that are evaluated via LLMs and enable users to perform various operations on multimodal data. Our benchmark introduces diversity across three key dimensions: scenarios, modalities, and operators. Included are scenarios ranging from movie review analysis to car damage detection. Within these scenarios, we cover different data modalities, including images, audio, and text. Finally, the queries involve a diverse set of operators, including semantic filters, joins, mappings, ranking, and classification operators. We evaluated our benchmark on three academic systems (LOTUS, Palimpzest, and ThalamusDB) and one industrial system, Google BigQuery. Although these results reflect a snapshot of systems under continuous development, our study offers crucial insights into their current strengths and weaknesses, illuminating promising directions for future research.

Zitieren

APA

Lao, J., Zimmerer, A., Ovcharenko, O., Cong, T., Russo, M., Vitagliano, G., Cochez, M., Özcan, F., Gupta, G., Hottelier, T., Jagadish, H. V., Kissel, K., Schelter, S., Kipf, A., & Trummer, I. (2026). SemBench: A Benchmark for Semantic Query Processing Engines. Proceedings of the VLDB Endowment, 19(8), 1754-1767. https://doi.org/10.14778/3811243.3811249

Vancouver

Lao J, Zimmerer A, Ovcharenko O, Cong T, Russo M, Vitagliano G, et al. SemBench: A Benchmark for Semantic Query Processing Engines. Proceedings of the VLDB Endowment. 2026;19(8):1754-1767. doi:10.14778/3811243.3811249

BibTeX
@article{Lao2026,
  author = {Lao, Jiale and Zimmerer, Andreas and Ovcharenko, Olga and Cong, Tianji and Russo, Matthew and Vitagliano, Gerardo and Cochez, Michael and Özcan, Fatma and Gupta, Gautam and Hottelier, Thibaud and Jagadish, H. V. and Kissel, Kris and Schelter, Sebastian and Kipf, Andreas and Trummer, Immanuel},
  title = {SemBench: A Benchmark for Semantic Query Processing Engines},
  journal = {Proceedings of the VLDB Endowment},
  year = {2026},
  volume = {19},
  number = {8},
  pages = {1754--1767},
  doi = {10.14778/3811243.3811249}
}
Kostenlos starten →