Benchmarking large language model performance in generating and assessing radiology objective structured clinical examination
Radiology Advances · 04.11.2026 · 3 (6)
DOI: 10.1093/radadv/umag039 ↗Kurz gefasst
Fachartikel in der Zeitschrift Radiology Advances, online erschienen am 12.09.2026. Verfasst von Ankush Ankush und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Radiology, Wake Forest University School of Medicine, Winston-Salem, NC 27157, United States. Die Arbeit nennt 27 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
Background
High-quality radiology assessment questions are essential for education competency evaluation but labor-intensive to create.
Purpose
To compare four large language models (LLMs) in generating and evaluating radiology objective structured clinical examination (OSCE)–style questions and responses. Materials and Methods Fifty Radiopaedia cases across 10 subspecialties were used to generate 200 radiology OSCE-style sessions (3 questions per session) using 4 LLMs: GPT-4o (4O), Llama 3-70b (LM), Claude 3.5 Sonnet (CL), and Gemini 1.5 Flash (GM). Three expert radiologists blindly evaluated sessions for clarity, clinical relevance, option accuracy, assessment accuracy, and feedback quality on 5-point Likert scales, with difficulty rated on a 3-point categorical scale. Generalized estimating equations compared model performance, accounting for rater correlation. The primary accuracy metric was Top Two Box Accuracy (all raters ≥4); secondary metrics included Top Box Accuracy, Average Score ≥4 Accuracy, and Perfect 5 Accuracy (all raters = 5). Final rankings were determined via Borda count.
Results
Inter-rater reliability ranged from fair to good agreement (Gwet’s AC2: 0.39-0.73 across the 5 retained metrics). Generalized estimating equations analysis showed significant performance variations (P < .05) across metrics. Final Borda scores: 4O (18.5), LM (14.5), CL (12.0), GM (5.0). Top Box Accuracy was high (70.7%-100%). For the primary metric Top Two Box Accuracy, 4O showed significantly higher odds of achieving high-quality scores vs. GM for clarity (odds ratio [OR] 4.52, P < .001) and overall assessment accuracy (OR 3.78, P < .001). Under the strictest threshold (Perfect 5 Accuracy), 4O led in composite Question General Accuracy (52.7% vs. 32.7% LM, 21.3% CL, 1.3% GM). Qualitative failure modes included distractor ambiguity, conceptual repetition, and information leakage.
Conclusion
This study provides a benchmark of LLM performance for radiology OSCE-style content generation and evaluation during a specific snapshot of artificial intelligence development (August 2024). 4O demonstrated the highest relative performance, though none consistently achieved expert-level quality. Although LLMs can augment radiology education, expert review remains mandatory for high-stakes assessment.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Radiology, Wake Forest University School of Medicine, Winston-Salem, NC 27157, United States.
- Department of Periodontics, Virginia Commonwealth University School of Dentistry, Richmond, VA 23298, United States.
- Department of Biostatistics and Data Science, Wake Forest University School of Medicine, Winston-Salem, NC 27157, United States.
- Department of Radiology and Interventional Radiology, Altnagelvin Hospital, Londonderry, Northern Ireland BT47 6SB, United Kingdom.
- Department of Radiology, UT Southwestern Medical Center, Dallas, TX 75390, United States.
- Department of Radiodiagnosis, Ram Krishna Medical College Hospital and Research Centre, Bhopal, MP 462042, India.
ORCID-Profile: Ankush Ankush, Samriddhi Burman, Sydney Smith, Vivek Gupta, Sitaram Barath, Monika Puranik, Janardhana Ponnatapura
Zitieren
APA
Ankush, A., Burman, S., Smith, S., Gupta, V., Barath, S., Puranik, M., & Ponnatapura, J. (2026). Benchmarking large language model performance in generating and assessing radiology objective structured clinical examination. Radiology Advances, 3(6). https://doi.org/10.1093/radadv/umag039
Vancouver
Ankush A, Burman S, Smith S, Gupta V, Barath S, Puranik M, et al. Benchmarking large language model performance in generating and assessing radiology objective structured clinical examination. Radiology Advances. 2026;3(6). doi:10.1093/radadv/umag039
BibTeX
@article{Ankush2026,
author = {Ankush, Ankush and Burman, Samriddhi and Smith, Sydney and Gupta, Vivek and Barath, Sitaram and Puranik, Monika and Ponnatapura, Janardhana},
title = {Benchmarking large language model performance in generating and assessing radiology objective structured clinical examination},
journal = {Radiology Advances},
year = {2026},
volume = {3},
number = {6},
doi = {10.1093/radadv/umag039}
}
EUVIMED