In-Situ Eval: A Modular Framework for Custom and Real-Time RAG Benchmarking
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (48) · 41586-41588
DOI: 10.1609/aaai.v40i48.42348 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Ritvik Garimella und 3 weiteren Autor:innen.
Abstract (Originalsprache)
Retrieval-Augmented Generation (RAG) has become the standard approach for integrating domain knowledge into Large Language Models (LLMs). However, fair comparison of RAG pipelines remains difficult: data preparation is often ad hoc, subsampling methods are opaque, parameters vary across implementations, and evaluation is fragmented. We present In-Situ Eval, a unified and reproducible framework that operationalizes the full RAG pipeline with configurable subsampling strategies and both RAG-specific and generic evaluation metrics. The platform supports two execution modes: an offline Dataset mode for evaluating precomputed outputs, and a live Retrieval mode for benchmarking RAG variants with state-of-the-art LLMs. Users can flexibly select datasets, retrieval techniques, models, and metrics, enabling side-by-side comparisons, ablations, and targeted analyses. This holistic approach reduces computational costs, clarifies the impact of subsampling techniques, and provides actionable insights for real-world deployments. By facilitating transparent, customizable, and interactive benchmarking, In-Situ Eval empowers both researchers and practitioners to make informed decisions in adapting RAG pipelines to domain-specific needs.
Zitieren
APA
Garimella, R., Roy, K., Shyalika, C., & Sheth, A. (2026). In-Situ Eval: A Modular Framework for Custom and Real-Time RAG Benchmarking. Proceedings of the AAAI Conference on Artificial Intelligence, 40(48), 41586-41588. https://doi.org/10.1609/aaai.v40i48.42348
Vancouver
Garimella R, Roy K, Shyalika C, Sheth A. In-Situ Eval: A Modular Framework for Custom and Real-Time RAG Benchmarking. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(48):41586-41588. doi:10.1609/aaai.v40i48.42348
BibTeX
@article{Garimella2026,
author = {Garimella, Ritvik and Roy, Kaushik and Shyalika, Chathurangi and Sheth, Amit},
title = {In-Situ Eval: A Modular Framework for Custom and Real-Time RAG Benchmarking},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {48},
pages = {41586--41588},
doi = {10.1609/aaai.v40i48.42348}
}
EUVIMED