Test-Retest Reliability of Large Language Models and Their Concordance With Human Experts in Rubric-Based Scoring of Clinical Summary Statements: Experimental Comparative Study
JMIR Medical Education · 21.10.2026 · 12 · e94666-e94666
DOI: 10.2196/94666 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift JMIR Medical Education, online erschienen am 21.10.2026. Verfasst von Patrick Stadler und 5 weiteren Autor:innen. Beteiligte Einrichtung: Charité - University Medical Center Berlin, Dieter Scheffner Center for Medical Education and Educational Research, Dean's Office for Study Affairs, …. Hauptthemen laut MEDLINE: Large Language Models, Educational Measurement. Die Arbeit nennt 56 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Clinical summary statements convert unstructured patient information into a concise format. They are widely used in virtual patients (VPs) to train medical students in diagnostic reasoning. Most VP systems provide little individualized feedback on the quality of learners’ summaries, and earlier automated approaches have required complex natural language processing, limiting their scalability. Large language models (LLMs) may offer a more flexible, rubric-guided way to score summary statements and support feedback. Still, their test-retest reliability and concordance with expert ratings in this context remain unclear.
Objective
This study aims to evaluate the test-retest reliability—specifically, whether the models produce stable ratings across repetitions—of LLM-generated medical summary statement ratings, and to measure their concordance with human expert assessments across different prompting strategies.
Methods
We conducted an experimental comparative study using 122 rubric-scored summary statements (English and German) from 6 internal medicine VP cases previously rated by 2 expert raters. Statements were evaluated along 4 ternary components (semantic qualifiers, transformation, narrowing, and global rating), and 2 binary components (factual accuracy and person). Four LLMs (Anthropic’s Claude, OpenAI’s GPT-4o, Mistral AI’s Mistral Large, and Mistral AI’s Mistral 7B) were each run 5 times under 4 prompting strategies (zero-shot, zero-shot with chain-of-thought [CoT], few-shot, and few-shot+CoT). The test-retest reliability was assessed with intraclass correlation coefficients (ICCs), Fleiss kappa (κ), and concordance with human ratings with accuracy (%) and chance-corrected interrater agreement (Gwet AC1 and AC2).
Results
Large closed-weights models (Claude, GPT-4o, and Mistral Large) showed high test-retest reliability (ICC≥0.92; Fleiss κ=0.59‐1.00). Claude was essentially deterministic (ICC and Fleiss κ≥0.99); GPT-4o and Mistral Large reached ICC≥0.97 and Fleiss κ≥0.80 except under zero-shot. Mistral 7B, a small open-weights model, had high ICCs (≥0.88) but low Fleiss κ (0.32‐0.52). Few-shot+CoT prompting achieved the best concordance with human ratings for all the models (means across runs and rubric components), peaking for Mistral Large (accuracy 77%; Gwet AC1 and AC2 0.85, 95% CI 0.83‐0.87), followed by Claude (71%; 0.77, 0.74‐0.81), GPT-4o (67%; 0.72, 0.68‐0.77), and Mistral 7B (60%; 0.64, 0.56‐0.73). Concordance was higher for the binary rubric components (person and factual accuracy) than for the ternary components. Extreme deviations and invalid outputs were rare but more frequent in ternary components and the smaller local model.
Conclusions
Large, general-purpose LLMs can function as reliable, rubric-guided raters of VP-based clinical summary statements, achieving high concordance with expert ratings under clearly defined criteria in binary rubric components and when using few-shot with CoT prompting. The performance on more nuanced, ternary reasoning dimensions was more variable. LLMs can support scalable, rubric-based formative feedback on clinical reasoning within VP systems but should complement rather than replace human judgment.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Charité - University Medical Center Berlin, Dieter Scheffner Center for Medical Education and Educational Research, Dean's Office for Study Affairs, Charitéplatz 1, Berlin, 10117, Germany, 49 30450576207.
- Faculty of Health Sciences Brandenburg, Institute of Research in Health Sciences Education, Brandenburg Medical School, Neuruppin, Germany.
Zitieren
APA
Stadler, P., Bucichowski, P., Holzhausen, Y., Hege, I., Peters, H., & Kaminski, J. J. (2026). Test-Retest Reliability of Large Language Models and Their Concordance With Human Experts in Rubric-Based Scoring of Clinical Summary Statements: Experimental Comparative Study. JMIR Medical Education, 12, e94666-e94666. https://doi.org/10.2196/94666
Vancouver
Stadler P, Bucichowski P, Holzhausen Y, Hege I, Peters H, Kaminski JJ. Test-Retest Reliability of Large Language Models and Their Concordance With Human Experts in Rubric-Based Scoring of Clinical Summary Statements: Experimental Comparative Study. JMIR Medical Education. 2026;12:e94666-e94666. doi:10.2196/94666
BibTeX
@article{Stadler2026,
author = {Stadler, Patrick and Bucichowski, Piotr and Holzhausen, Ylva and Hege, Inga and Peters, Harm and Kaminski, Julius Josef},
title = {Test-Retest Reliability of Large Language Models and Their Concordance With Human Experts in Rubric-Based Scoring of Clinical Summary Statements: Experimental Comparative Study},
journal = {JMIR Medical Education},
year = {2026},
volume = {12},
pages = {e94666--e94666},
doi = {10.2196/94666}
}
EUVIMED