A Self-Controlled Benchmark of Retrieval-Augmented Generation for Large Language Models on Clinical Guideline Questions
Diagnostics · 04.08.2026 · 16 (15) · 2456
DOI: 10.3390/diagnostics16152456 ↗Kurz gefasst
Leitlinie in der Zeitschrift Diagnostics, online erschienen am 04.08.2026. Verfasst von Andreas Vollmer und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Oral and Maxillofacial Plastic Surgery, University Hospital of Wuerzburg, 97070 Wuerzburg, Germany. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 30 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Background/Objectives: Large language models (LLMs) show promise for clinical decision support, yet their accuracy in interpreting specialized medical guidelines remains uncertain. Retrieval-augmented generation (RAG) may enhance performance by grounding responses in authoritative knowledge bases. This study aimed to compare the accuracy, comprehensiveness, and safety of RAG-enhanced versus standard LLMs for answering clinical questions derived from the German S3 guideline for oral cavity carcinoma.
Methods
We conducted a prospective, single-blind benchmark study evaluating six LLMs: one RAG-enhanced model (Custom GPT with guideline access), one consensus-based model (ConsensusGPT), and four standard models (DeepSeek-V3.2, Mistral Small 3.2, Qwen3-Next-80B, GPT-OSS-120B). Fifty clinical questions covering 17 guideline domains were presented to each model three times, yielding 900 evaluations. Three expert reviewers assessed responses using 5-point Likert scales for accuracy, comprehensiveness, and clarity, under a single-blind procedure, the effectiveness of which was tested by a pre-specified manipulation check. We then ran a paired within-model experiment in which each base model was queried with and without guideline access through a transparent, openly released retrieval pipeline, and scored every response with a condition-blind automated judge alongside deterministic retrieval metrics computed from the logs. Secondary outcomes included hallucination rates and guideline citation behavior. Inter-rater reliability was assessed using intraclass correlation coefficients (ICCs).
Results
In a paired within-model design that held each base model fixed, adding transparent guideline retrieval improved accuracy—significantly in the three weaker open-weight models (Mistral, Qwen3, and GPT-OSS) and directionally in the already-strong DeepSeek and GPT-5 bases. Because a pre-specified blinding check found that experts could still identify retrieval-augmented answers with 98.5% accuracy, we anchored causal interpretation on measures that do not depend on the human raters, ranked by their independence: deterministic, log-derived retrieval metrics first, and then an automated, condition-blind LLM judge, whose agreement with the experts (Spearman ρ = 0.81, 95.7% within-one agreement) establishes shared calibration rather than independence from their bias. Deterministically from the retrieval logs, citation groundedness rose from 0% to 51–89% and retrieval recall@5 was 92%. On the judge, content-level hallucination fell from 42% to 4% and accuracy rose by a pooled +0.64 points (95% CI 0.47–0.80); the accuracy gain persisted after adjustment for response length (+0.48, 95% CI 0.22–0.73), which retrieval shortened rather than lengthened. The accuracy gain was large for weaker base models and small or non-significant for already-strong ones, whereas the hallucination and auditability gains were consistent across all models. The human ratings reproduced the judge’s accuracy effect (+0.61, 95% CI 0.49–0.74), and GPT-5 run through the transparent pipeline showed no significant difference from the proprietary Custom GPT (judge accuracy 4.48 vs. 4.58).
Conclusions
Guideline retrieval yields a reproducible, largely base-independent improvement in the safety and auditability of LLM answers to clinical guideline questions, with accuracy gains concentrated in weaker base models. Because retrieval-augmented answers are recognizable to experts, rigorous evaluation should rely on rater-independent measures, and residual hallucination continues to require human oversight.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Oral and Maxillofacial Plastic Surgery, University Hospital of Wuerzburg, 97070 Wuerzburg, Germany.
- Department of Oral, Maxillofacial and Facial Plastic Surgery, Medical Faculty and University Hospital Duesseldorf, Heinrich-Heine-University Duesseldorf, 40225 Duesseldorf, Germany.
- Department of Oral and Maxillofacial Surgery, Tuebingen University Hospital, Osianderstrasse 2-8, 72076 Tuebingen, Germany.
ORCID-Profile: Andreas Vollmer, Lara Schorn, Christoph Sproll, Babak Saravi
Zitieren
APA
Vollmer, A., Schorn, L., Schrader, F., Kübler, N., Sproll, C., Vollmer, M., Singh, D. D., & Saravi, B. (2026). A Self-Controlled Benchmark of Retrieval-Augmented Generation for Large Language Models on Clinical Guideline Questions. Diagnostics, 16(15), 2456. https://doi.org/10.3390/diagnostics16152456
Vancouver
Vollmer A, Schorn L, Schrader F, Kübler N, Sproll C, Vollmer M, et al. A Self-Controlled Benchmark of Retrieval-Augmented Generation for Large Language Models on Clinical Guideline Questions. Diagnostics. 2026;16(15):2456. doi:10.3390/diagnostics16152456
BibTeX
@article{Vollmer2026,
author = {Vollmer, Andreas and Schorn, Lara and Schrader, Felix and Kübler, Norbert and Sproll, Christoph and Vollmer, Michael and Singh, Daman Deep and Saravi, Babak},
title = {A Self-Controlled Benchmark of Retrieval-Augmented Generation for Large Language Models on Clinical Guideline Questions},
journal = {Diagnostics},
year = {2026},
volume = {16},
number = {15},
pages = {2456},
doi = {10.3390/diagnostics16152456}
}
EUVIMED