Comparative analysis of large language model and faculty rater scoring of medical student documentation in objective structured clinical examinations
Vergleichende Analyse der Bewertung medizinischer Student:innen-Dokumentation durch ein Large Language Model und Fakultätsrater in objektiven strukturierten klinischen Prüfungen
Academic Medicine · 24.10.2026
DOI: 10.1093/acamed/wvag336 ↗Auf einen Blick
Ein kommerzielles LLM bewertete OSCE-Notizen von Medizinstudierenden konsistent. Die absoluten Scores lagen jedoch signifikant unter den Fakultätsbewertungen.
Fragestellung
Kann ein kommerziell verfügbares LLM OSCE-Post-Encounter-Notizen mit einer institutionellen Rubrik ohne modellspezifisches Training bewerten und wie stimmt es mit Fakultätswertungen überein?
Methode
433 Notizen aus 7 standardisierten OSCE-Fällen an der University of Kentucky; Bewertung mit OpenAI GPT-4o mini per Prompt, ohne Fine-Tuning; Vergleich mit historischen Fakultätsnoten.
Ergebnisse
GPT-NE: ICC 0,94; GPT-E: ICC 0,76. LLM-Scores lagen unter Fakultätswerten: -1,12 (95 %-KI -1,17 bis -1,08) für GPT-NE und -0,65 (95 %-KI -0,69 bis -0,61) für GPT-E; Spearman r = 0,43–0,53.
Schlussfolgerung der Autor:innen
Das LLM zeigte Potenzial für die automatisierte OSCE-Notenbewertung durch hohe interne Konsistenz und ähnliche Rangfolge der Lernenden wie Fakultätsrater. Abweichungen von den absoluten Fakultätswerten erfordern weitere Anpassung.
Kernaussagen
- 433 OSCE-Notizen aus 7 Fällen wurden mit GPT-4o mini bewertet
- Interne Konsistenz hoch: ICC 0,94; 0,76
- Scores unter Fakultätswerten, Rangfolge moderat erhalten
Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.
Kurz gefasst
Fachartikel in der Zeitschrift Academic Medicine, online erschienen am 24.10.2026. Verfasst von Christopher J Nash und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Emergency Medicine, Duke University Hospital, Durham, North Carolina, USA. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Purpose
To evaluate whether a commercially available large language model (LLM) can score medical student post-encounter notes from Objective Structured Clinical Examinations (OSCEs) using an existing institutional rubric without task-specific model training and to characterize internal consistency and alignment with faculty scores using absolute agreement and rank-order association metrics.
Method
The dataset included 433 medical student notes from 7 standardized OSCE cases at the University of Kentucky. In November 2024, an LLM (OpenAI GPT-4o mini, via API) scored each note using a prompt-based workflow (no model fine-tuning) under two approaches: numerical score only (“GPT-NE”, 30 iterations per note) and numerical score with rationale (“GPT-E”, 4 iterations per note). The authors compared the LLM’s performance against historical faculty grades using the same rubric, evaluating the reliability of grading methodologies and inter-modal agreement.
Results
The LLM demonstrated high internal consistency for GPT-NE (intraclass correlation (ICC) = 0.94) and moderate-to-high consistency for GPT-E (ICC = 0.76). LLM scores were significantly lower than faculty scores (mean differences on a 1–4 scale: -1.12 GPT-NE, 95% CI -1.17 to -1.08; -0.65 GPT-E, 95% CI -0.69 to -0.61, both P < .001). Spearman rank-order correlations (r = 0.43–0.53) indicate moderate preservation of relative student ranking despite differences in absolute scores.
Conclusions
The LLM demonstrated significant potential for automating OSCE note grading given its internal consistency and similar ranking of learners compared with faculty raters. Automation of grading could yield substantial time savings; however, discrepancies with absolute faculty scores underscore the need for further refinement of rubric alignment and grading methodologies.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Emergency Medicine, Duke University Hospital, Durham, North Carolina, USA.
- Department of Clinical Science, Kaiser Permanente Bernard J. Tyson School of Medicine, Pasadena, California, USA.
- Department of Pediatrics, Children's Hospital of Richmond at Virginia Commonwealth University, Richmond, Virginia, USA.
- Department of Learning Health Sciences, University of Michigan Medical School, Ann Arbor, Michigan, USA.
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA.
- Department of Emergency Medicine, University of Kentucky College of Medicine, Lexington, Kentucky, USA.
ORCID-Profile: Christopher J Nash, Candace Y Pau, Nayef Chahin, Larry D Gruppen, Jennifer R Kogan, Tama S Thé
Zitieren
APA
Nash, C. J., Pau, C. Y., Chahin, N., Gruppen, L. D., Kogan, J. R., & Thé, T. S. (2026). Comparative analysis of large language model and faculty rater scoring of medical student documentation in objective structured clinical examinations. Academic Medicine. https://doi.org/10.1093/acamed/wvag336
Vancouver
Nash CJ, Pau CY, Chahin N, Gruppen LD, Kogan JR, Thé TS. Comparative analysis of large language model and faculty rater scoring of medical student documentation in objective structured clinical examinations. Academic Medicine. 2026. doi:10.1093/acamed/wvag336
BibTeX
@article{Nash2026,
author = {Nash, Christopher J and Pau, Candace Y and Chahin, Nayef and Gruppen, Larry D and Kogan, Jennifer R and Thé, Tama S},
title = {Comparative analysis of large language model and faculty rater scoring of medical student documentation in objective structured clinical examinations},
journal = {Academic Medicine},
year = {2026},
doi = {10.1093/acamed/wvag336}
}
EUVIMED