Reasoning‐optimised large language models reach near‐expert accuracy on board‐style orthopaedic exams: A multi‐model comparison on 702 multiple‐choice questions
Knee Surgery, Sports Traumatology, Arthroscopy · 2026-02 · 34 (2) · 752-762
DOI: 10.1002/ksa.70222 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Knee Surgery, Sports Traumatology, Arthroscopy, online erschienen am 17.12.2025. Verfasst von Pedro Diniz und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Orthopaedic Surgery, Centre Hospitalier Universitaire Brugmann, Brussels, Belgium. Hauptthemen laut MEDLINE: Orthopedics, Educational Measurement, Language, Clinical Competence. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 39 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Purpose
The purpose of this study was to compare the accuracy, calibration, reproducibility and operating cost of seven large language models (LLMs)—including four newer models capable of using advanced reasoning techniques to analyse complex medical information and generate accurate responses—on text‐only orthopaedic multiple‐choice questions (MCQs) and to quantify gains over GPT‐4.
Methods
From Orthobullets, 702 unique, non‐image MCQs (drawn from AAOS Self‐Assessment Examinations, Self‐Assessment‐Based Questions and Orthopaedic In Training Examination‐Based Questions banks) were extracted. Each question was submitted to the following LLMs: OpenAI o3, Anthropic Claude Sonnet 4, Claude Opus 4 (with/without ‘Extended Thinking’) and Google Gemini 2.5 Pro. Additionally, OpenAI's GPT‐4, GPT‐4o and the open‐weight Gemma 3 27B served as comparators. The primary outcome was overall accuracy. The secondary outcomes were topic and difficulty‐stratified accuracy, calibration (expected calibration error [ECE] and Brier score), reproducibility (flip rate on a retest question subset), latency, token use and cost. Statistical tests included paired McNemar, Cochran Q , ordinal logistic regression and Fleiss κ (Bonferroni‐adjusted α = 0.05).
Results
GPT‐4 achieved 69.7% accuracy (95% CI = 66.2–72.9). All four reasoning‐optimised models scored ≥14 percentage points higher ( p < 3.3 × 10 −15 ); OpenAI o3 led with 93.6% (95% CI = 91.5–95.2), which represents a 34% relative error reduction. Accuracy tended to decline with question difficulty, yet the reasoning advantage persisted in every difficulty stratum. Claude Opus 4 showed the best calibration (ECE = 0.023), while GPT‐4 exhibited overconfidence (ECE = 0.215). All models except Gemma 3 27B exhibited non‐zero flip rates. Median query time: 0.9 s (Gemma) to 15.9 s (Gemini 2.5 Pro). Cost: 0 to 29.9 USD per 1000 queries.
Conclusions
Reasoning‐optimised LLMs now answer text‐based orthopaedic exam questions with high accuracy and substantially better confidence calibration than earlier models. However, persistent stochasticity and large latency‐cost disparities may limit clinical deployment. Level of Evidence N/A.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Orthopaedic Surgery, Centre Hospitalier Universitaire Brugmann, Brussels, Belgium.
- Department of Orthopaedic Surgery, Hôpital Universitaire de Bruxelles - Hôpital Erasme, Brussels, Belgium.
- Department of Bioengineering and iBB - Institute for Bioengineering and Biosciences, Instituto Superior Técnico, Universidade de Lisboa, Lisbon, Portugal.
- Division of Orthopaedic Surgery, Department of Medicine of Sensory and Motor Organs, Faculty of Medicine, University of Miyazaki, Miyazaki, Japan.
- Department of Orthopaedic Surgery, Balgrist University Hospital, Zürich, Switzerland.
- Orthopaedic Department, Centro Hospitalar Póvoa de Varzim, Vila do Conde, Portugal.
ORCID-Profile: Pedro Diniz
Zitieren
APA
Diniz, P., Yokoe, T., Öttl, F. C., Pereira, H., Henriques, R., & Samuelsson, K. (2026). Reasoning‐optimised large language models reach near‐expert accuracy on board‐style orthopaedic exams: A multi‐model comparison on 702 multiple‐choice questions. Knee Surgery, Sports Traumatology, Arthroscopy, 34(2), 752-762. https://doi.org/10.1002/ksa.70222
Vancouver
Diniz P, Yokoe T, Öttl FC, Pereira H, Henriques R, Samuelsson K. Reasoning‐optimised large language models reach near‐expert accuracy on board‐style orthopaedic exams: A multi‐model comparison on 702 multiple‐choice questions. Knee Surgery, Sports Traumatology, Arthroscopy. 2026;34(2):752-762. doi:10.1002/ksa.70222
BibTeX
@article{Diniz2026,
author = {Diniz, Pedro and Yokoe, Takuji and Öttl, Felix C and Pereira, Hélder and Henriques, Rui and Samuelsson, Kristian},
title = {Reasoning‐optimised large language models reach near‐expert accuracy on board‐style orthopaedic exams: A multi‐model comparison on 702 multiple‐choice questions},
journal = {Knee Surgery, Sports Traumatology, Arthroscopy},
year = {2026},
volume = {34},
number = {2},
pages = {752--762},
doi = {10.1002/ksa.70222}
}
EUVIMED