Multi-reader, multi-model benchmark of large language models for modified Outerbridge cartilage grading from knee MRI reports
Skeletal Radiology · 2026-12 · 55 (12) · 3217-3228
DOI: 10.1007/s00256-026-05342-9 ↗Kurz gefasst
Fachartikel in der Zeitschrift Skeletal Radiology, online erschienen am 18.08.2026. Verfasst von Anna H Zhao und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, 15 Francis St, Boston, MA, 02115, USA. Hauptthemen laut MEDLINE: Large Language Models, Magnetic Resonance Imaging, Knee Joint, Cartilage, Articular. Die Arbeit nennt 23 Literaturangaben. Gefördert von: Brigham and Women’s Hospital.
Abstract (Originalsprache)
OBJECTIVE: To describe a reproducible framework for bulk large language model (LLM)-based extraction of structured cartilage-lesion data from knee MRI reports and to benchmark seven LLM configurations against multiple radiologists using the modified Outerbridge classification. MATERIALS AND METHODS: In this IRB-approved retrospective study, 100 non-contrast knee MRI reports (January 2019 to January 2025) were randomly selected from 66,479 eligible examinations and independently graded by five readers (four fellowship-trained musculoskeletal radiologists with 6-21 years of post-fellowship experience and one fourth-year resident) and seven LLM configurations, comprising six Azure OpenAI deployments (GPT-4.1, GPT-5.1-mini, GPT-5.3, GPT-5.4, GPT-5.4-mini, GPT-5.4-nano) and one locally hosted open-weight model (Qwen2.5-32B-Instruct), across a fixed 20-surface anatomic taxonomy. Pairwise Cohen quadratic-weighted κ was computed at compartment and surface levels. Bootstrap 95% confidence intervals (CIs) and p-values on the paired difference between each rater's mean κ and the human-human reference were derived by case-level cluster resampling (B = 1000). RESULTS: Mean human-human κ was 0.94 (95% CI 0.93, 0.95) at the compartment level and 0.80 (95% CI 0.78, 0.83) at the surface level. Three Azure deployments-GPT-5.4 (surface κ = 0.83), GPT-5.3 (κ = 0.80), and GPT-5.1-mini (κ = 0.77)-had 95% CIs on the difference from the human reference that included zero. Four models had CIs entirely below zero, indicating agreement significantly below the human reference: GPT-4.1, GPT-5.4-mini, Qwen2.5-32B, and GPT-5.4-nano (all p ≤ .02). CONCLUSION: Three flagship OpenAI deployments achieved cartilage grading agreement indistinguishable from human inter-rater variability, while cost-optimized and open-weight variants performed measurably below.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, 15 Francis St, Boston, MA, 02115, USA.
- Harvard Medical School, 25 Shattuck St, Boston, MA, 02114, USA.
- Department of Radiology, Division of Musculoskeletal Imaging, Brigham and Women's Hospital, Harvard Medical School, 15 Francis St, Boston, MA, 02115, USA.
- Department of Radiology, Division of Musculoskeletal Imaging, Brigham and Women's Hospital, Harvard Medical School, 15 Francis St, Boston, MA, 02115, USA. jmandell@bwh.harvard.edu.
Förderung
- Brigham and Women’s Hospital
Zitieren
APA
Zhao, A. H., Roux, E. L., Wieschhoff, G., Bahouth, S., Smith, S. E., Wu, J. S., & Mandell, J. C. (2026). Multi-reader, multi-model benchmark of large language models for modified Outerbridge cartilage grading from knee MRI reports. Skeletal Radiology, 55(12), 3217-3228. https://doi.org/10.1007/s00256-026-05342-9
Vancouver
Zhao AH, Roux EL, Wieschhoff G, Bahouth S, Smith SE, Wu JS, et al. Multi-reader, multi-model benchmark of large language models for modified Outerbridge cartilage grading from knee MRI reports. Skeletal Radiology. 2026;55(12):3217-3228. doi:10.1007/s00256-026-05342-9
BibTeX
@article{Zhao2026,
author = {Zhao, Anna H and Roux, Elizabeth L and Wieschhoff, Ged and Bahouth, Sara and Smith, Stacy E and Wu, Jim S and Mandell, Jacob C},
title = {Multi-reader, multi-model benchmark of large language models for modified Outerbridge cartilage grading from knee MRI reports},
journal = {Skeletal Radiology},
year = {2026},
volume = {55},
number = {12},
pages = {3217--3228},
doi = {10.1007/s00256-026-05342-9}
}
EUVIMED