EUVIMEDEUVIMED entdecken
Crossref

RAGPerf: An End-to-End Benchmarking Framework for Retrieval-Augmented Generation Systems

Shaobo Li, Yirui Zhou, Yuan Xu, Kevin Chen, Daniel Waddington, Swaminathan Sundararaman, Hubertus Franke, Jian Huang

Proceedings of the VLDB Endowment · 01.07.2026 · 19 (11) · 3689-3703

DOI: 10.14778/3836663.3836718 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the VLDB Endowment, online erschienen am 04.09.2026. Verfasst von Shaobo Li und 7 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 117 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We present the design and implementation of a RAG-based AI system benchmarking (RAGPerf) framework for characterizing the system behaviors of RAG pipelines. To facilitate detailed profiling and fine-grained performance analysis, RAGPerf decouples the RAG workflow into several modular components — embedding, indexing, retrieval, reranking, and generation. RAGPerf offers the flexibility for users to configure the core parameters of each component and examine their impact on the end-to-end query performance and quality. RAGPerf has a workload generator to model real-world scenarios by supporting diverse datasets (e.g., text, pdf, code, and audio), different retrieval and update ratios, and query distributions. RAGPerf also supports different embedding models, major vector databases such as LanceDB, Milvus, Qdrant, Chroma, and Elasticsearch, as well as different LLMs for content generation. It automates the collection of performance metrics (i.e., end-to-end query throughput, host/GPU memory footprint, and CPU/GPU utilization) and accuracy metrics (i.e., context recall, query accuracy, and factual consistency). We demonstrate the capabilities of RAGPerf through a comprehensive set of experiments and open source its codebase at GitHub. Our evaluation shows that RAGPerf incurs negligible performance overhead.

Zitieren

APA

Li, S., Zhou, Y., Xu, Y., Chen, K., Waddington, D., Sundararaman, S., Franke, H., & Huang, J. (2026). RAGPerf: An End-to-End Benchmarking Framework for Retrieval-Augmented Generation Systems. Proceedings of the VLDB Endowment, 19(11), 3689-3703. https://doi.org/10.14778/3836663.3836718

Vancouver

Li S, Zhou Y, Xu Y, Chen K, Waddington D, Sundararaman S, et al. RAGPerf: An End-to-End Benchmarking Framework for Retrieval-Augmented Generation Systems. Proceedings of the VLDB Endowment. 2026;19(11):3689-3703. doi:10.14778/3836663.3836718

BibTeX
@article{Li2026,
  author = {Li, Shaobo and Zhou, Yirui and Xu, Yuan and Chen, Kevin and Waddington, Daniel and Sundararaman, Swaminathan and Franke, Hubertus and Huang, Jian},
  title = {RAGPerf: An End-to-End Benchmarking Framework for Retrieval-Augmented Generation Systems},
  journal = {Proceedings of the VLDB Endowment},
  year = {2026},
  volume = {19},
  number = {11},
  pages = {3689--3703},
  doi = {10.14778/3836663.3836718}
}
Kostenlos starten →