Performance of GPT-5 and Gemini 2.5 Pro on the Orthopaedic In-Training Examination
Orthopedic Reviews · 17.04.2026 · 18
DOI: 10.52965/001c.160184 ↗Kurz gefasst
Fachartikel in der Zeitschrift Orthopedic Reviews, online erschienen am 17.04.2026. Verfasst von Le Duc Huy und 7 weiteren Autor:innen. Beteiligte Einrichtung: College of Health Sciences VinUniversity. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 18 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Previous studies evaluating large language models (LLMs) on the Orthopaedic In-Training Examination (OITE) have primarily focused on earlier-generation models and single-pass accuracy. These investigations did not assess newer multimodal systems such as GPT-5 and Gemini 2.5 Pro, nor did they examine the reasoning quality underlying model responses or the consistency of outputs across repeated trials. As LLMs are increasingly used as educational tools, a more comprehensive evaluation framework is needed to assess not only correctness but also reliability and explanatory validity on specialty-specific, image-rich examinations.
Methods
We conducted a controlled, parallel evaluation of GPT-5 and Gemini 2.5 Pro using 412 OITE-style questions from the 2023–2024 examination cycle obtained via an institutional AAOS ResStudy subscription. Primary outcomes included overall and subspecialty-specific accuracy. Secondary analyses evaluated explanatory quality, error-pattern classification, response consistency across repeated trials, and performance stratified by imaging burden. Paired accuracy was compared using McNemar’s exact test.
Results
Gemini 2.5 Pro demonstrated higher overall accuracy than GPT-5 on the 2023–2024 OITE question set (81.1% vs 76.0), with both models exceeding published PGY-5 resident benchmarks. Accuracy declined significantly with questions containing images (74.2% vs 71.6%). Subspecialty performance varied widely, with accuracy ranging from 42.9% to 94.1% for GPT-5 and from 57.1% to 95.8% for Gemini, and both models performing poorest in Hand and Wrist questions. Among incorrect responses, faulty reasoning accounted for 52.5% of GPT-5 errors, whereas stem misinterpretation was the predominant error for Gemini (43.6%). Incorrect or partially correct explanations accompanied 45.4% of GPT-5 and 41.7% of Gemini responses. Consistency testing showed high reproducibility (fully consistent responses: 88% for GPT-5 and 84% for Gemini), with all inconsistent outputs occurring in image-containing questions.
Conclusions
GPT-5 and Gemini 2.5 Pro demonstrate strong performance on recent OITE content, exceeding prior LLM benchmarks; however, persistent limitations in multimodal reasoning, explanatory reliability, and response consistency indicate that high accuracy alone does not ensure dependable clinical reasoning, underscoring the need for cautious educational use.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- College of Health Sciences VinUniversity.
- Orthopaedics & Sports Medicine Center Vinmec International Hospital.
- Department of General Orthopedics 108 Military Central Hospital.
Zitieren
APA
Huy, L. D., Anh, L. N., Quang, M. H., Phuc, L. H., Trung, N. D., Thang, V. D., Nam, L. H., & Dung, T. T. (2026). Performance of GPT-5 and Gemini 2.5 Pro on the Orthopaedic In-Training Examination. Orthopedic Reviews, 18. https://doi.org/10.52965/001c.160184
Vancouver
Huy LD, Anh LN, Quang MH, Phuc LH, Trung ND, Thang VD, et al. Performance of GPT-5 and Gemini 2.5 Pro on the Orthopaedic In-Training Examination. Orthopedic Reviews. 2026;18. doi:10.52965/001c.160184
BibTeX
@article{Huy2026,
author = {Huy, Le Duc and Anh, Luong Nhat and Quang, Mai Huy and Phuc, Luu Huu and Trung, Nguyen Duc and Thang, Vu Duc and Nam, Le Hoai and Dung, Tran Trung},
title = {Performance of GPT-5 and Gemini 2.5 Pro on the Orthopaedic In-Training Examination},
journal = {Orthopedic Reviews},
year = {2026},
volume = {18},
doi = {10.52965/001c.160184}
}
EUVIMED