Large Language Model Responses to Common Otolaryngological Questions: Evaluating Accuracy, Appropriateness, Readability, and Hallucinations
Otolaryngology–Head and Neck Surgery · 2026
DOI: 10.1002/ohn.70309 ↗Kurz gefasst
Beobachtungsstudie in der Zeitschrift Otolaryngology–Head and Neck Surgery, 2026. Verfasst von Mitali Sakharkar und 5 weiteren Autor:innen. Beteiligte Einrichtung: Boston University Chobanian and Avedisian School of Medicine, Boston, Massachusetts, USA. Hauptthemen laut MEDLINE: Large Language Models, Comprehension, Otolaryngology, Hallucinations. Die Arbeit nennt 23 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Objectives
Artificial Intelligence (AI) is increasingly integrated into medicine, including otolaryngology. However, concerns remain regarding the accuracy of generated content and the tendency of large language models (LLMs) to fabricate references. This study evaluates the accuracy, appropriateness, readability, and hallucination of references in 2 prevalent large language models, ChatGPT and Claude, in response to common otolaryngological questions. Study Design Prospective observational study. Settings Academic tertiary care center.
Methods
Thirty‐six otolaryngologic questions were individually entered into ChatGPT 4.0 Plus and CLAUDE in separate sessions, with explicit instructions to avoid utilizing previous memories. To assess reproducibility, each query was submitted twice. Two otolaryngologists independently rated the accuracy of responses. Readability was evaluated using the Flesch Reading Ease (FRE) score. Reference hallucinations were assessed by analyzing the reference validity and relevance.
Results
ChatGPT and CLAUDE had an FRE of 47 and 25.2 out of 100, respectively. For patient readability, ChatGPT scored a 3.60 while Claude scored a 4.68 out of 5. Claude scored slightly higher on accuracy, receiving a score of 4.42 out of 5 while ChatGPT received a 3.81. Both models hallucinated at least half of their references, with some citations irrelevant or incorrectly formatted. Thematic analysis revealed frequent vagueness, poor clinical prioritization, and excessive jargon across both models.
Conclusion
Both ChatGPT and CLAUDE often produced partially inaccurate, jargon‐filled responses and failed to consistently provide valid references when answering common otolaryngologic patient questions. Our results highlight the need for better understanding and regulation of LLM limitations in clinical and patient‐facing applications.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Boston University Chobanian and Avedisian School of Medicine, Boston, Massachusetts, USA.
- Boston University Department of Otolaryngology, Boston, Massachusetts, USA.
ORCID-Profile: Mitali Sakharkar, Sarah Chang, Jessica Levi
Zitieren
APA
Sakharkar, M., Jalihal, P., Chang, S., Patel, P., Chavez, M., & Levi, J. (2026). Large Language Model Responses to Common Otolaryngological Questions: Evaluating Accuracy, Appropriateness, Readability, and Hallucinations. Otolaryngology–Head and Neck Surgery. https://doi.org/10.1002/ohn.70309
Vancouver
Sakharkar M, Jalihal P, Chang S, Patel P, Chavez M, Levi J. Large Language Model Responses to Common Otolaryngological Questions: Evaluating Accuracy, Appropriateness, Readability, and Hallucinations. Otolaryngology–Head and Neck Surgery. 2026. doi:10.1002/ohn.70309
BibTeX
@article{Sakharkar2026,
author = {Sakharkar, Mitali and Jalihal, Parth and Chang, Sarah and Patel, Prachi and Chavez, Melissa and Levi, Jessica},
title = {Large Language Model Responses to Common Otolaryngological Questions: Evaluating Accuracy, Appropriateness, Readability, and Hallucinations},
journal = {Otolaryngology–Head and Neck Surgery},
year = {2026},
doi = {10.1002/ohn.70309}
}
EUVIMED