EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Why NoSQL Benchmarks Fail in AI Era: A Practical Validity-Aware Workflow for Trustworthy Data Store Evaluation

Kiran Fahd, Sitalakshmi Venkatraman, Sazia Parvin

Systems · 10.11.2026 · 14 (11) · 1441

DOI: 10.3390/systems14111441 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Systems, online erschienen am 10.11.2026. Verfasst von Kiran Fahd und 2 weiteren Autor:innen. Die Arbeit nennt 43 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Selecting a data-management subsystem is a consequential decision in data-intensive applications, including AI pipelines that rely on low-latency storage. Such decisions are often informed by lightweight, application-specific benchmarks whose measurement validity is rarely examined. This paper presents a validity-aware benchmarking workflow for comparing NoSQL data stores under application-representative CRUD workloads. The workflow incorporates a final validation stage that checks whether measured operations exhibit the expected behaviour before performance comparisons are made. The approach is demonstrated using a key-value store (Aerospike) and a document store (MongoDB). The validity checks identified two operations that performed no actual database work. MongoDB’s idiomatic read operation returned a cursor that was never iterated, causing reads to appear approximately 130 times faster for single-record workloads and 1000 times faster for bulk workloads than equivalent Aerospike reads. In addition, a MongoDB bulk update matched no documents because update keys were constructed incorrectly. Further validity threats included a missing secondary index, unequal bulk workloads, and a first-run warmup effect. Consequently, the remaining write measurements are reported descriptively, together with confidence intervals and run-level results. The study does not establish which engine delivers superior performance. Instead, it demonstrates how seemingly reasonable application-specific benchmarks can generate misleading conclusions and argues for an explicit validity-assessment stage, supported by a threat model, before benchmark results are used to inform subsystem selection and architectural decision-making.

Zitieren

APA

Fahd, K., Venkatraman, S., & Parvin, S. (2026). Why NoSQL Benchmarks Fail in AI Era: A Practical Validity-Aware Workflow for Trustworthy Data Store Evaluation. Systems, 14(11), 1441. https://doi.org/10.3390/systems14111441

Vancouver

Fahd K, Venkatraman S, Parvin S. Why NoSQL Benchmarks Fail in AI Era: A Practical Validity-Aware Workflow for Trustworthy Data Store Evaluation. Systems. 2026;14(11):1441. doi:10.3390/systems14111441

BibTeX
@article{Fahd2026,
  author = {Fahd, Kiran and Venkatraman, Sitalakshmi and Parvin, Sazia},
  title = {Why NoSQL Benchmarks Fail in AI Era: A Practical Validity-Aware Workflow for Trustworthy Data Store Evaluation},
  journal = {Systems},
  year = {2026},
  volume = {14},
  number = {11},
  pages = {1441},
  doi = {10.3390/systems14111441}
}
Kostenlos starten →