EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Benchmarking large language model performance in generating and assessing radiology objective structured clinical examination

Ankush Ankush iD, Samriddhi Burman iD, Sydney Smith iD, Vivek Gupta iD, Sitaram Barath iD, Monika Puranik iD, Janardhana Ponnatapura iD

Radiology Advances · 04.11.2026 · 3 (6)

DOI: 10.1093/radadv/umag039 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Radiology Advances, online erschienen am 12.09.2026. Verfasst von Ankush Ankush und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Radiology, Wake Forest University School of Medicine, Winston-Salem, NC 27157, United States. Die Arbeit nennt 27 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

High-quality radiology assessment questions are essential for education competency evaluation but labor-intensive to create.

Purpose

To compare four large language models (LLMs) in generating and evaluating radiology objective structured clinical examination (OSCE)–style questions and responses. Materials and Methods Fifty Radiopaedia cases across 10 subspecialties were used to generate 200 radiology OSCE-style sessions (3 questions per session) using 4 LLMs: GPT-4o (4O), Llama 3-70b (LM), Claude 3.5 Sonnet (CL), and Gemini 1.5 Flash (GM). Three expert radiologists blindly evaluated sessions for clarity, clinical relevance, option accuracy, assessment accuracy, and feedback quality on 5-point Likert scales, with difficulty rated on a 3-point categorical scale. Generalized estimating equations compared model performance, accounting for rater correlation. The primary accuracy metric was Top Two Box Accuracy (all raters ≥4); secondary metrics included Top Box Accuracy, Average Score ≥4 Accuracy, and Perfect 5 Accuracy (all raters = 5). Final rankings were determined via Borda count.

Results

Inter-rater reliability ranged from fair to good agreement (Gwet’s AC2: 0.39-0.73 across the 5 retained metrics). Generalized estimating equations analysis showed significant performance variations (P < .05) across metrics. Final Borda scores: 4O (18.5), LM (14.5), CL (12.0), GM (5.0). Top Box Accuracy was high (70.7%-100%). For the primary metric Top Two Box Accuracy, 4O showed significantly higher odds of achieving high-quality scores vs. GM for clarity (odds ratio [OR] 4.52, P < .001) and overall assessment accuracy (OR 3.78, P < .001). Under the strictest threshold (Perfect 5 Accuracy), 4O led in composite Question General Accuracy (52.7% vs. 32.7% LM, 21.3% CL, 1.3% GM). Qualitative failure modes included distractor ambiguity, conceptual repetition, and information leakage.

Conclusion

This study provides a benchmark of LLM performance for radiology OSCE-style content generation and evaluation during a specific snapshot of artificial intelligence development (August 2024). 4O demonstrated the highest relative performance, though none consistently achieved expert-level quality. Although LLMs can augment radiology education, expert review remains mandatory for high-stakes assessment.

Schlagwörter

Stichwörter der Autor:innen

artificial intelligencebenchmarkingclinical competenceeducational measurementlarge language modelsradiology

Autor:innen und Institutionen

  • Department of Radiology, Wake Forest University School of Medicine, Winston-Salem, NC 27157, United States.
  • Department of Periodontics, Virginia Commonwealth University School of Dentistry, Richmond, VA 23298, United States.
  • Department of Biostatistics and Data Science, Wake Forest University School of Medicine, Winston-Salem, NC 27157, United States.
  • Department of Radiology and Interventional Radiology, Altnagelvin Hospital, Londonderry, Northern Ireland BT47 6SB, United Kingdom.
  • Department of Radiology, UT Southwestern Medical Center, Dallas, TX 75390, United States.
  • Department of Radiodiagnosis, Ram Krishna Medical College Hospital and Research Centre, Bhopal, MP 462042, India.

ORCID-Profile: Ankush Ankush, Samriddhi Burman, Sydney Smith, Vivek Gupta, Sitaram Barath, Monika Puranik, Janardhana Ponnatapura

Zitieren

APA

Ankush, A., Burman, S., Smith, S., Gupta, V., Barath, S., Puranik, M., & Ponnatapura, J. (2026). Benchmarking large language model performance in generating and assessing radiology objective structured clinical examination. Radiology Advances, 3(6). https://doi.org/10.1093/radadv/umag039

Vancouver

Ankush A, Burman S, Smith S, Gupta V, Barath S, Puranik M, et al. Benchmarking large language model performance in generating and assessing radiology objective structured clinical examination. Radiology Advances. 2026;3(6). doi:10.1093/radadv/umag039

BibTeX
@article{Ankush2026,
  author = {Ankush, Ankush and Burman, Samriddhi and Smith, Sydney and Gupta, Vivek and Barath, Sitaram and Puranik, Monika and Ponnatapura, Janardhana},
  title = {Benchmarking large language model performance in generating and assessing radiology objective structured clinical examination},
  journal = {Radiology Advances},
  year = {2026},
  volume = {3},
  number = {6},
  doi = {10.1093/radadv/umag039}
}
Kostenlos starten →