Comparative Assessment of Otolaryngology Knowledge Among Large Language Models
The Laryngoscope · 2025
DOI: 10.1002/lary.31781 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift The Laryngoscope, 2025. Verfasst von Dante J Merlino und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Otolaryngology-Head and Neck Surgery, Mayo Clinic, Rochester, Minnesota, U.S.A. Hauptthemen laut MEDLINE: Otolaryngology, Educational Measurement, Clinical Competence. Die Arbeit wurde bisher 17-mal zitiert (Crossref) und nennt 33 Literaturangaben.
Abstract (Originalsprache)
Objective
The purpose of this study was to evaluate the performance of advanced large language models from OpenAI (GPT‐3.5 and GPT‐4), Google (PaLM2 and MedPaLM), and an open source model from Meta (Llama3:70b) in answering clinical test multiple choice questions in the field of otolaryngology—head and neck surgery.
Methods
A dataset of 4566 otolaryngology questions was used; each model was provided a standardized prompt followed by a question. One hundred questions that were answered incorrectly by all models were further interrogated to gain insight into the causes of incorrect answers.
Results
GPT4 was the most accurate, correctly answering 3520 of 4566 questions (77.1%). MedPaLM correctly answered 3223 of 4566 (70.6%) questions, while llama3:70b, GPT3.5, and PaLM2 were correct on 3052 of 4566 (66.8%), 2672 of 4566 (58.5%), and 2583 of 4566 (56.5%) questions. Three hundred and sixty‐nine questions were answered incorrectly by all models. Prompts to provide reasoning improved accuracy in all models: GPT4 changed from incorrect to correct answer 31% of the time, while GPT3.5, Llama3, PaLM2, and MedPaLM corrected their responses 25%, 18%, 19%, and 17% of the time, respectively.
Conclusion
Large language models vary in their understanding of otolaryngology‐specific clinical knowledge. OpenAI's GPT4 has a strong understanding of core concepts as well as detailed information in the field of otolaryngology. Its baseline understanding in this field makes it well‐suited to serve in roles related to head and neck surgery education provided that the appropriate precautions are taken and potential limitations are understood. Level of Evidence NA Laryngoscope , 135:629–634, 2025
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Otolaryngology-Head and Neck Surgery, Mayo Clinic, Rochester, Minnesota, U.S.A.
- The Center for Plastic Surgery at Castle Rock, Castle Rock, Colorado, U.S.A.
- Department of Otolaryngology-Head and Neck Surgery, University of Kansas Medical Center, Kansas City, Kansas, U.S.A.
- Department of Neurologic Surgery, Mayo Clinic, Rochester, Minnesota, U.S.A.
ORCID-Profile: Dante J Merlino, Kathryn M Van Abel, Matthew L Carlson
Zitieren
APA
Merlino, D. J., Brufau, S. R., Saieed, G., Van Abel, K. M., Price, D. L., Archibald, D. J., Ator, G. A., & Carlson, M. L. (2025). Comparative Assessment of Otolaryngology Knowledge Among Large Language Models. The Laryngoscope. https://doi.org/10.1002/lary.31781
Vancouver
Merlino DJ, Brufau SR, Saieed G, Van Abel KM, Price DL, Archibald DJ, et al. Comparative Assessment of Otolaryngology Knowledge Among Large Language Models. The Laryngoscope. 2025. doi:10.1002/lary.31781
BibTeX
@article{Merlino2025,
author = {Merlino, Dante J and Brufau, Santiago R and Saieed, George and Van Abel, Kathryn M and Price, Daniel L and Archibald, David J and Ator, Gregory A and Carlson, Matthew L},
title = {Comparative Assessment of Otolaryngology Knowledge Among Large Language Models},
journal = {The Laryngoscope},
year = {2025},
doi = {10.1002/lary.31781}
}
EUVIMED