EUVIMEDEUVIMED entdecken
Vergleichsstudie

Comparative Assessment of Otolaryngology Knowledge Among Large Language Models

Dante J Merlino iD, Santiago R Brufau, George Saieed, Kathryn M Van Abel iD, Daniel L Price, David J Archibald, Gregory A Ator, Matthew L Carlson iD

The Laryngoscope · 2025

DOI: 10.1002/lary.31781 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift The Laryngoscope, 2025. Verfasst von Dante J Merlino und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Otolaryngology-Head and Neck Surgery, Mayo Clinic, Rochester, Minnesota, U.S.A. Hauptthemen laut MEDLINE: Otolaryngology, Educational Measurement, Clinical Competence. Die Arbeit wurde bisher 17-mal zitiert (Crossref) und nennt 33 Literaturangaben.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

The purpose of this study was to evaluate the performance of advanced large language models from OpenAI (GPT‐3.5 and GPT‐4), Google (PaLM2 and MedPaLM), and an open source model from Meta (Llama3:70b) in answering clinical test multiple choice questions in the field of otolaryngology—head and neck surgery.

Methods

A dataset of 4566 otolaryngology questions was used; each model was provided a standardized prompt followed by a question. One hundred questions that were answered incorrectly by all models were further interrogated to gain insight into the causes of incorrect answers.

Results

GPT4 was the most accurate, correctly answering 3520 of 4566 questions (77.1%). MedPaLM correctly answered 3223 of 4566 (70.6%) questions, while llama3:70b, GPT3.5, and PaLM2 were correct on 3052 of 4566 (66.8%), 2672 of 4566 (58.5%), and 2583 of 4566 (56.5%) questions. Three hundred and sixty‐nine questions were answered incorrectly by all models. Prompts to provide reasoning improved accuracy in all models: GPT4 changed from incorrect to correct answer 31% of the time, while GPT3.5, Llama3, PaLM2, and MedPaLM corrected their responses 25%, 18%, 19%, and 17% of the time, respectively.

Conclusion

Large language models vary in their understanding of otolaryngology‐specific clinical knowledge. OpenAI's GPT4 has a strong understanding of core concepts as well as detailed information in the field of otolaryngology. Its baseline understanding in this field makes it well‐suited to serve in roles related to head and neck surgery education provided that the appropriate precautions are taken and potential limitations are understood. Level of Evidence NA Laryngoscope , 135:629–634, 2025

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

OtolaryngologyEducational MeasurementClinical CompetenceHumansLarge Language Models

Stichwörter der Autor:innen

AIENTartificial intelligenceeducationlarge language modelsotolaryngology

Autor:innen und Institutionen

  • Department of Otolaryngology-Head and Neck Surgery, Mayo Clinic, Rochester, Minnesota, U.S.A.
  • The Center for Plastic Surgery at Castle Rock, Castle Rock, Colorado, U.S.A.
  • Department of Otolaryngology-Head and Neck Surgery, University of Kansas Medical Center, Kansas City, Kansas, U.S.A.
  • Department of Neurologic Surgery, Mayo Clinic, Rochester, Minnesota, U.S.A.

ORCID-Profile: Dante J Merlino, Kathryn M Van Abel, Matthew L Carlson

Zitieren

APA

Merlino, D. J., Brufau, S. R., Saieed, G., Van Abel, K. M., Price, D. L., Archibald, D. J., Ator, G. A., & Carlson, M. L. (2025). Comparative Assessment of Otolaryngology Knowledge Among Large Language Models. The Laryngoscope. https://doi.org/10.1002/lary.31781

Vancouver

Merlino DJ, Brufau SR, Saieed G, Van Abel KM, Price DL, Archibald DJ, et al. Comparative Assessment of Otolaryngology Knowledge Among Large Language Models. The Laryngoscope. 2025. doi:10.1002/lary.31781

BibTeX
@article{Merlino2025,
  author = {Merlino, Dante J and Brufau, Santiago R and Saieed, George and Van Abel, Kathryn M and Price, Daniel L and Archibald, David J and Ator, Gregory A and Carlson, Matthew L},
  title = {Comparative Assessment of Otolaryngology Knowledge Among Large Language Models},
  journal = {The Laryngoscope},
  year = {2025},
  doi = {10.1002/lary.31781}
}
Kostenlos starten →