EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Evidence on Demand: OpenEvidence versus Established LLMs on the Orthopaedic In-Training Examination

Asim A. Khan, Shaan S. Lalvani, Sam Pourarbab, Lord J. Hyeamang, Ryan M. Lew, Zaamin B. Hussain, Gregory P. Nicholson, Grant E. Garrigues

Journal of Orthopaedic Experience & Innovation · 26.09.2026 · 7 (2)

DOI: 10.60118/001c.162882 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Orthopaedic Experience & Innovation, online erschienen am 26.09.2026. Verfasst von Asim A. Khan und 7 weiteren Autor:innen. Die Arbeit nennt 23 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models (LLMs) are increasingly used in medical education, yet head-to-head comparisons of contemporary multimodal and clinically oriented (retrieval-grounded) models on orthopedic in-service examinations remain limited.

Methods

A cross-sectional comparative study was conducted using 265 multiple-choice questions from the publicly available 2014 Orthopaedic In-Training Examination (OITE). Questions were categorized by AAOS subspecialty and by cognitive taxonomy (T1 recall, T2 interpretation, T3 reasoning) and stratified into text-only (n=111) versus image-based items (n=164). ChatGPT-5 Plus and Gemini 2.5 Pro were evaluated on the full dataset, while OpenEvidence was evaluated on the text-only subset. Accuracy was scored against the official answer key and comparisons were done using McNemar and Cochran Q tests.

Results

Overall accuracy on the full 2014 OITE was 80.0% for Gemini 2.5 Pro and 78.1% for ChatGPT-5, with no statistically significant difference (p=0.59). Across the text-only subset, Gemini 2.5 Pro and OpenEvidence each scored 84.7% versus 81.1% for ChatGPT-5 (p=0.71), and OpenEvidence demonstrated strong T3 reasoning performance (82.7%). Subspecialty performance varied without statistically significant differences. LLMs demonstrated highest accuracy in basic science and the largest numerical discrepancy was in Oncology (Gemini 77.3% vs ChatGPT-5 54.5%).

Conclusion

A retrieval-grounded, evidence based LLM (OpenEvidence) performed comparably to multimodal LLMs on text-based orthopaedic in-training examination questions at or above senior resident PGY-5 benchmark-level performance with no significant differences in overall accuracy. These findings suggest that OpenEvidence may provide educational utility comparable to general-purpose models while offering source transparency.

Zitieren

APA

Khan, A. A., Lalvani, S. S., Pourarbab, S., Hyeamang, L. J., Lew, R. M., Hussain, Z. B., Nicholson, G. P., & Garrigues, G. E. (2026). Evidence on Demand: OpenEvidence versus Established LLMs on the Orthopaedic In-Training Examination. Journal of Orthopaedic Experience & Innovation, 7(2). https://doi.org/10.60118/001c.162882

Vancouver

Khan AA, Lalvani SS, Pourarbab S, Hyeamang LJ, Lew RM, Hussain ZB, et al. Evidence on Demand: OpenEvidence versus Established LLMs on the Orthopaedic In-Training Examination. Journal of Orthopaedic Experience & Innovation. 2026;7(2). doi:10.60118/001c.162882

BibTeX
@article{Khan2026,
  author = {Khan, Asim A. and Lalvani, Shaan S. and Pourarbab, Sam and Hyeamang, Lord J. and Lew, Ryan M. and Hussain, Zaamin B. and Nicholson, Gregory P. and Garrigues, Grant E.},
  title = {Evidence on Demand: OpenEvidence versus Established LLMs on the Orthopaedic In-Training Examination},
  journal = {Journal of Orthopaedic Experience & Innovation},
  year = {2026},
  volume = {7},
  number = {2},
  doi = {10.60118/001c.162882}
}
Kostenlos starten →