Comparative evaluation of LLMs in orthopedic surgery
Journal of Orthopaedic Reports · 2026-04 · 5 (2) · 100728
DOI: 10.1016/j.jorep.2025.100728 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Orthopaedic Reports, 2026-04. Verfasst von Gnaneswar Chundi und 5 weiteren Autor:innen. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 28 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
AIMS/OBJECTIVES This study aimed to evaluate the performance of leading large language models (LLMs) in orthopedic surgery, with a focus on diagnostic accuracy, radiographic interpretation, subspecialty-specific performance, consistency, and gender bias. Unlike previous investigations that focused solely on ChatGPT, we assessed Claude-3-Sonnet, GPT-4o, Gemini-1.5, and Meta’s LLaMA Vision-Instruct models against Orthopaedic In-Training Examination (OITE) questions. METHODS We tested each LLM on 2,906 multiple-choice questions from the OITE question bank. Questions were categorized by subspecialty and by presence of images. Models provided answer choices, confidence scores (1–4), and justifications. Each question was administered three times to evaluate consistency. Statistical analyses included Z-tests, t-tests, ANOVA, chi-square tests, and logistic regression. Rasch transformation enabled comparison to PGY-1 and PGY-5 resident performance. Gender bias was evaluated based on performance differences across gender-specific cases. RESULTS GPT-4o achieved the highest accuracy (72.8%), outperforming all other models. Performance improved with larger model sizes across all vendors. All models showed diminished accuracy on image-based questions (mean 10.9% lower, p
Zitieren
APA
Chundi, G., Dawar, A., Sarwar, S., Prasad, S., Vosbikian, M., & Ahmed, I. (2026). Comparative evaluation of LLMs in orthopedic surgery. Journal of Orthopaedic Reports, 5(2), 100728. https://doi.org/10.1016/j.jorep.2025.100728
Vancouver
Chundi G, Dawar A, Sarwar S, Prasad S, Vosbikian M, Ahmed I. Comparative evaluation of LLMs in orthopedic surgery. Journal of Orthopaedic Reports. 2026;5(2):100728. doi:10.1016/j.jorep.2025.100728
BibTeX
@article{Chundi2026,
author = {Chundi, Gnaneswar and Dawar, Abhiram and Sarwar, Syed and Prasad, Sanjiv and Vosbikian, Michael and Ahmed, Irfan},
title = {Comparative evaluation of LLMs in orthopedic surgery},
journal = {Journal of Orthopaedic Reports},
year = {2026},
volume = {5},
number = {2},
pages = {100728},
doi = {10.1016/j.jorep.2025.100728}
}
EUVIMED