EUVIMEDEUVIMED entdecken
BeobachtungsstudieOpen Access

Large Language Model Responses to Common Otolaryngological Questions: Evaluating Accuracy, Appropriateness, Readability, and Hallucinations

Mitali Sakharkar iD, Parth Jalihal, Sarah Chang iD, Prachi Patel, Melissa Chavez, Jessica Levi iD

Otolaryngology–Head and Neck Surgery · 2026

DOI: 10.1002/ohn.70309 ↗

Kurz gefasst

Beobachtungsstudie in der Zeitschrift Otolaryngology–Head and Neck Surgery, 2026. Verfasst von Mitali Sakharkar und 5 weiteren Autor:innen. Beteiligte Einrichtung: Boston University Chobanian and Avedisian School of Medicine, Boston, Massachusetts, USA. Hauptthemen laut MEDLINE: Large Language Models, Comprehension, Otolaryngology, Hallucinations. Die Arbeit nennt 23 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

Artificial Intelligence (AI) is increasingly integrated into medicine, including otolaryngology. However, concerns remain regarding the accuracy of generated content and the tendency of large language models (LLMs) to fabricate references. This study evaluates the accuracy, appropriateness, readability, and hallucination of references in 2 prevalent large language models, ChatGPT and Claude, in response to common otolaryngological questions. Study Design Prospective observational study. Settings Academic tertiary care center.

Methods

Thirty‐six otolaryngologic questions were individually entered into ChatGPT 4.0 Plus and CLAUDE in separate sessions, with explicit instructions to avoid utilizing previous memories. To assess reproducibility, each query was submitted twice. Two otolaryngologists independently rated the accuracy of responses. Readability was evaluated using the Flesch Reading Ease (FRE) score. Reference hallucinations were assessed by analyzing the reference validity and relevance.

Results

ChatGPT and CLAUDE had an FRE of 47 and 25.2 out of 100, respectively. For patient readability, ChatGPT scored a 3.60 while Claude scored a 4.68 out of 5. Claude scored slightly higher on accuracy, receiving a score of 4.42 out of 5 while ChatGPT received a 3.81. Both models hallucinated at least half of their references, with some citations irrelevant or incorrectly formatted. Thematic analysis revealed frequent vagueness, poor clinical prioritization, and excessive jargon across both models.

Conclusion

Both ChatGPT and CLAUDE often produced partially inaccurate, jargon‐filled responses and failed to consistently provide valid references when answering common otolaryngologic patient questions. Our results highlight the need for better understanding and regulation of LLM limitations in clinical and patient‐facing applications.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsComprehensionOtolaryngologyHallucinationsHumansProspective StudiesReproducibility of ResultsGenerative Artificial Intelligence

Stichwörter der Autor:innen

Artificial IntelligenceChatGPTLarge Language Modelsmedical misinformationpatient resources

Autor:innen und Institutionen

  • Boston University Chobanian and Avedisian School of Medicine, Boston, Massachusetts, USA.
  • Boston University Department of Otolaryngology, Boston, Massachusetts, USA.

ORCID-Profile: Mitali Sakharkar, Sarah Chang, Jessica Levi

Zitieren

APA

Sakharkar, M., Jalihal, P., Chang, S., Patel, P., Chavez, M., & Levi, J. (2026). Large Language Model Responses to Common Otolaryngological Questions: Evaluating Accuracy, Appropriateness, Readability, and Hallucinations. Otolaryngology–Head and Neck Surgery. https://doi.org/10.1002/ohn.70309

Vancouver

Sakharkar M, Jalihal P, Chang S, Patel P, Chavez M, Levi J. Large Language Model Responses to Common Otolaryngological Questions: Evaluating Accuracy, Appropriateness, Readability, and Hallucinations. Otolaryngology–Head and Neck Surgery. 2026. doi:10.1002/ohn.70309

BibTeX
@article{Sakharkar2026,
  author = {Sakharkar, Mitali and Jalihal, Parth and Chang, Sarah and Patel, Prachi and Chavez, Melissa and Levi, Jessica},
  title = {Large Language Model Responses to Common Otolaryngological Questions: Evaluating Accuracy, Appropriateness, Readability, and Hallucinations},
  journal = {Otolaryngology–Head and Neck Surgery},
  year = {2026},
  doi = {10.1002/ohn.70309}
}
Kostenlos starten →