EUVIMEDSkopri EUVIMED
BeobachtungsstudieOpen access

Large Language Model Responses to Common Otolaryngological Questions: Evaluating Accuracy, Appropriateness, Readability, and Hallucinations

Mitali Sakharkar iD, Parth Jalihal, Sarah Chang iD, Prachi Patel, Melissa Chavez, Jessica Levi iD

Otolaryngology–Head and Neck Surgery · 2026

DOI: 10.1002/ohn.70309 ↗

At a glance

Beobachtungsstudie in Otolaryngology–Head and Neck Surgery, 2026. Written by Mitali Sakharkar and 5 co-authors. Institution: Boston University Chobanian and Avedisian School of Medicine, Boston, Massachusetts, USA. Major MEDLINE topics: Large Language Models, Comprehension, Otolaryngology, Hallucinations. 23 references. The full text is freely available in PubMed Central.

AstrattSubjectsDettalji biblijografiċiCiteRelated studies

Abstract (original language)

Objectives

Artificial Intelligence (AI) is increasingly integrated into medicine, including otolaryngology. However, concerns remain regarding the accuracy of generated content and the tendency of large language models (LLMs) to fabricate references. This study evaluates the accuracy, appropriateness, readability, and hallucination of references in 2 prevalent large language models, ChatGPT and Claude, in response to common otolaryngological questions. Study Design Prospective observational study. Settings Academic tertiary care center.

Methods

Thirty‐six otolaryngologic questions were individually entered into ChatGPT 4.0 Plus and CLAUDE in separate sessions, with explicit instructions to avoid utilizing previous memories. To assess reproducibility, each query was submitted twice. Two otolaryngologists independently rated the accuracy of responses. Readability was evaluated using the Flesch Reading Ease (FRE) score. Reference hallucinations were assessed by analyzing the reference validity and relevance.

Results

ChatGPT and CLAUDE had an FRE of 47 and 25.2 out of 100, respectively. For patient readability, ChatGPT scored a 3.60 while Claude scored a 4.68 out of 5. Claude scored slightly higher on accuracy, receiving a score of 4.42 out of 5 while ChatGPT received a 3.81. Both models hallucinated at least half of their references, with some citations irrelevant or incorrectly formatted. Thematic analysis revealed frequent vagueness, poor clinical prioritization, and excessive jargon across both models.

Conclusion

Both ChatGPT and CLAUDE often produced partially inaccurate, jargon‐filled responses and failed to consistently provide valid references when answering common otolaryngologic patient questions. Our results highlight the need for better understanding and regulation of LLM limitations in clinical and patient‐facing applications.

Subjects

MeSH terms (MEDLINE)

Large Language ModelsComprehensionOtolaryngologyHallucinationsHumansProspective StudiesReproducibility of ResultsGenerative Artificial Intelligence

Author keywords

Artificial IntelligenceChatGPTLarge Language Modelsmedical misinformationpatient resources

Authors and institutions

  • Boston University Chobanian and Avedisian School of Medicine, Boston, Massachusetts, USA.
  • Boston University Department of Otolaryngology, Boston, Massachusetts, USA.

ORCID profiles: Mitali Sakharkar, Sarah Chang, Jessica Levi

Cite

APA

Sakharkar, M., Jalihal, P., Chang, S., Patel, P., Chavez, M., & Levi, J. (2026). Large Language Model Responses to Common Otolaryngological Questions: Evaluating Accuracy, Appropriateness, Readability, and Hallucinations. Otolaryngology–Head and Neck Surgery. https://doi.org/10.1002/ohn.70309

Vancouver

Sakharkar M, Jalihal P, Chang S, Patel P, Chavez M, Levi J. Large Language Model Responses to Common Otolaryngological Questions: Evaluating Accuracy, Appropriateness, Readability, and Hallucinations. Otolaryngology–Head and Neck Surgery. 2026. doi:10.1002/ohn.70309

BibTeX
@article{Sakharkar2026,
  author = {Sakharkar, Mitali and Jalihal, Parth and Chang, Sarah and Patel, Prachi and Chavez, Melissa and Levi, Jessica},
  title = {Large Language Model Responses to Common Otolaryngological Questions: Evaluating Accuracy, Appropriateness, Readability, and Hallucinations},
  journal = {Otolaryngology–Head and Neck Surgery},
  year = {2026},
  doi = {10.1002/ohn.70309}
}
Start for free →