Why NoSQL Benchmarks Fail in AI Era: A Practical Validity-Aware Workflow for Trustworthy Data Store Evaluation
Systems · 10.11.2026 · 14 (11) · 1441
DOI: 10.3390/systems14111441 ↗Kurz gefasst
Fachartikel in der Zeitschrift Systems, online erschienen am 10.11.2026. Verfasst von Kiran Fahd und 2 weiteren Autor:innen. Die Arbeit nennt 43 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Selecting a data-management subsystem is a consequential decision in data-intensive applications, including AI pipelines that rely on low-latency storage. Such decisions are often informed by lightweight, application-specific benchmarks whose measurement validity is rarely examined. This paper presents a validity-aware benchmarking workflow for comparing NoSQL data stores under application-representative CRUD workloads. The workflow incorporates a final validation stage that checks whether measured operations exhibit the expected behaviour before performance comparisons are made. The approach is demonstrated using a key-value store (Aerospike) and a document store (MongoDB). The validity checks identified two operations that performed no actual database work. MongoDB’s idiomatic read operation returned a cursor that was never iterated, causing reads to appear approximately 130 times faster for single-record workloads and 1000 times faster for bulk workloads than equivalent Aerospike reads. In addition, a MongoDB bulk update matched no documents because update keys were constructed incorrectly. Further validity threats included a missing secondary index, unequal bulk workloads, and a first-run warmup effect. Consequently, the remaining write measurements are reported descriptively, together with confidence intervals and run-level results. The study does not establish which engine delivers superior performance. Instead, it demonstrates how seemingly reasonable application-specific benchmarks can generate misleading conclusions and argues for an explicit validity-assessment stage, supported by a threat model, before benchmark results are used to inform subsystem selection and architectural decision-making.
Zitieren
APA
Fahd, K., Venkatraman, S., & Parvin, S. (2026). Why NoSQL Benchmarks Fail in AI Era: A Practical Validity-Aware Workflow for Trustworthy Data Store Evaluation. Systems, 14(11), 1441. https://doi.org/10.3390/systems14111441
Vancouver
Fahd K, Venkatraman S, Parvin S. Why NoSQL Benchmarks Fail in AI Era: A Practical Validity-Aware Workflow for Trustworthy Data Store Evaluation. Systems. 2026;14(11):1441. doi:10.3390/systems14111441
BibTeX
@article{Fahd2026,
author = {Fahd, Kiran and Venkatraman, Sitalakshmi and Parvin, Sazia},
title = {Why NoSQL Benchmarks Fail in AI Era: A Practical Validity-Aware Workflow for Trustworthy Data Store Evaluation},
journal = {Systems},
year = {2026},
volume = {14},
number = {11},
pages = {1441},
doi = {10.3390/systems14111441}
}
EUVIMED