Comparing Large Language Models' Performances on Otolaryngology Knowledge Assessment Questions
Applied Clinical Informatics · 01.03.2026 · 17 (02) · 194-203
DOI: 10.1055/a-2835-4634 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Applied Clinical Informatics, erschienen am 01.03.2026. Verfasst von Ryan Cook und 7 weiteren Autor:innen. Beteiligte Einrichtung: Albert Einstein College of Medicine, Bronx, New York, United States. Hauptthemen laut MEDLINE: Large Language Models, Otolaryngology, Educational Measurement, Knowledge. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 27 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
This study evaluates the performance of multiple large language models (LLMs) on specialized otolaryngology knowledge, comparing OpenAI's GPT-4 Turbo with 10 commercially available models to assess their potential utility in otolaryngology medical education. A total of 1,075 questions from OTO QUEST, the official self-assessment resource of the American Academy of Otolaryngology–Head and Neck Surgery, were administered to GPT-4 Turbo using a zero-shot approach. Accuracy was analyzed using logistic regression, adjusting for question difficulty, year, and subspecialty. Performance was subsequently compared with 10 other commercial models (including Claude-3.5-Sonnet, Gemini-1.5-Pro, and GPT-4o) using the 1,075 question dataset, evaluated with Cochran's Q test (p < 0.001) and McNemar's pairwise comparison. GPT-4 Turbo achieved an overall accuracy of 72.09% (95% confidence interval [CI]: 69.3–74.7%) across 1,075 questions. It performed best in Practice Management questions (odds ratio [OR] = 3.93, 95% CI: 1.12–13.73, p = 0.032) and declined in accuracy when faced with questions of moderate and hard difficulty (OR = 0.21, 95% CI: 0.16–0.29, p < 0.001 and OR = 0.04, 95% CI: 0.01–0.10, p < 0.001, respectively). In comparative analysis, Grok-3 ranked highest with 76.3% accuracy (95% CI: 73.6–78.7%), followed by Claude-3.5-sonnet (73.0%, 95% CI: 70.3–75.6%) and GPT-4o (69.9%, 95% CI: 67.1–72.5%), with GPT-4 Turbo application programming interface ranking fourth. This comprehensive model comparison reveals that while major commercial LLMs show promising capabilities in specialized medical knowledge assessment, they demonstrate an apparent accuracy plateau around 73 to 76%. These findings suggest current general-purpose LLMs may require specialized training approaches to advance beyond this performance threshold in medical domains.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Albert Einstein College of Medicine, Bronx, New York, United States.
- Department of Otolaryngology-Head and Neck Surgery, Thomas Jefferson University, Philadelphia, Pennsylvania, United States.
- Department of Otolaryngology-Head and Neck Surgery, Montefiore Einstein, Bronx, New York, United States.
- Department of Medicine, Montefiore Einstein, Bronx, New York, United States.
ORCID-Profile: Ryan Cook, Sunit P Jariwala
Zitieren
APA
Cook, R., Kahan, A., Scharfenberger, T., Tasoulas, J., Hawks-Ladds, N., Chouake, R., Jariwala, S. P., & Arora, S. (2026). Comparing Large Language Models' Performances on Otolaryngology Knowledge Assessment Questions. Applied Clinical Informatics, 17(02), 194-203. https://doi.org/10.1055/a-2835-4634
Vancouver
Cook R, Kahan A, Scharfenberger T, Tasoulas J, Hawks-Ladds N, Chouake R, et al. Comparing Large Language Models' Performances on Otolaryngology Knowledge Assessment Questions. Applied Clinical Informatics. 2026;17(02):194-203. doi:10.1055/a-2835-4634
BibTeX
@article{Cook2026,
author = {Cook, Ryan and Kahan, Abner and Scharfenberger, Thomas and Tasoulas, Jason and Hawks-Ladds, Noah and Chouake, Robert and Jariwala, Sunit P and Arora, Shitij},
title = {Comparing Large Language Models' Performances on Otolaryngology Knowledge Assessment Questions},
journal = {Applied Clinical Informatics},
year = {2026},
volume = {17},
number = {02},
pages = {194--203},
doi = {10.1055/a-2835-4634}
}
EUVIMED