EUVIMEDEUVIMED entdecken
Open Access

Performance of GPT-5 and Gemini 2.5 Pro on the Orthopaedic In-Training Examination

Le Duc Huy, Luong Nhat Anh, Mai Huy Quang, Luu Huu Phuc, Nguyen Duc Trung, Vu Duc Thang, Le Hoai Nam, Tran Trung Dung

Orthopedic Reviews · 17.04.2026 · 18

DOI: 10.52965/001c.160184 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Orthopedic Reviews, online erschienen am 17.04.2026. Verfasst von Le Duc Huy und 7 weiteren Autor:innen. Beteiligte Einrichtung: College of Health Sciences VinUniversity. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 18 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Previous studies evaluating large language models (LLMs) on the Orthopaedic In-Training Examination (OITE) have primarily focused on earlier-generation models and single-pass accuracy. These investigations did not assess newer multimodal systems such as GPT-5 and Gemini 2.5 Pro, nor did they examine the reasoning quality underlying model responses or the consistency of outputs across repeated trials. As LLMs are increasingly used as educational tools, a more comprehensive evaluation framework is needed to assess not only correctness but also reliability and explanatory validity on specialty-specific, image-rich examinations.

Methods

We conducted a controlled, parallel evaluation of GPT-5 and Gemini 2.5 Pro using 412 OITE-style questions from the 2023–2024 examination cycle obtained via an institutional AAOS ResStudy subscription. Primary outcomes included overall and subspecialty-specific accuracy. Secondary analyses evaluated explanatory quality, error-pattern classification, response consistency across repeated trials, and performance stratified by imaging burden. Paired accuracy was compared using McNemar’s exact test.

Results

Gemini 2.5 Pro demonstrated higher overall accuracy than GPT-5 on the 2023–2024 OITE question set (81.1% vs 76.0), with both models exceeding published PGY-5 resident benchmarks. Accuracy declined significantly with questions containing images (74.2% vs 71.6%). Subspecialty performance varied widely, with accuracy ranging from 42.9% to 94.1% for GPT-5 and from 57.1% to 95.8% for Gemini, and both models performing poorest in Hand and Wrist questions. Among incorrect responses, faulty reasoning accounted for 52.5% of GPT-5 errors, whereas stem misinterpretation was the predominant error for Gemini (43.6%). Incorrect or partially correct explanations accompanied 45.4% of GPT-5 and 41.7% of Gemini responses. Consistency testing showed high reproducibility (fully consistent responses: 88% for GPT-5 and 84% for Gemini), with all inconsistent outputs occurring in image-containing questions.

Conclusions

GPT-5 and Gemini 2.5 Pro demonstrate strong performance on recent OITE content, exceeding prior LLM benchmarks; however, persistent limitations in multimodal reasoning, explanatory reliability, and response consistency indicate that high accuracy alone does not ensure dependable clinical reasoning, underscoring the need for cautious educational use.

Schlagwörter

Stichwörter der Autor:innen

Artificial intelligenceChatGPTGeminiLarge language modelsOITE

Autor:innen und Institutionen

  • College of Health Sciences VinUniversity.
  • Orthopaedics & Sports Medicine Center Vinmec International Hospital.
  • Department of General Orthopedics 108 Military Central Hospital.

Zitieren

APA

Huy, L. D., Anh, L. N., Quang, M. H., Phuc, L. H., Trung, N. D., Thang, V. D., Nam, L. H., & Dung, T. T. (2026). Performance of GPT-5 and Gemini 2.5 Pro on the Orthopaedic In-Training Examination. Orthopedic Reviews, 18. https://doi.org/10.52965/001c.160184

Vancouver

Huy LD, Anh LN, Quang MH, Phuc LH, Trung ND, Thang VD, et al. Performance of GPT-5 and Gemini 2.5 Pro on the Orthopaedic In-Training Examination. Orthopedic Reviews. 2026;18. doi:10.52965/001c.160184

BibTeX
@article{Huy2026,
  author = {Huy, Le Duc and Anh, Luong Nhat and Quang, Mai Huy and Phuc, Luu Huu and Trung, Nguyen Duc and Thang, Vu Duc and Nam, Le Hoai and Dung, Tran Trung},
  title = {Performance of GPT-5 and Gemini 2.5 Pro on the Orthopaedic In-Training Examination},
  journal = {Orthopedic Reviews},
  year = {2026},
  volume = {18},
  doi = {10.52965/001c.160184}
}
Kostenlos starten →