EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access · CC BY 4.0

Comparative Analysis of Information Quality in Pediatric Otorhinolaryngology: Clinicians, Residents, and Large Language Models

Eleonora M C Trecca, Vito Carlo Alberto Caponio, Mario Turri-Zanoni, Antonella Miriam di Lullo iD, Michele Gaffuri, Jérôme R Lechien, Antonino Maniaci, Giuseppe Maruccio iD, Marella Reale, Irene Claudia Visconti, Virginia Dallari

Otolaryngology–Head and Neck Surgery · 2025-07 · 173 (1) · 228-236

DOI: 10.1002/ohn.1225 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift Otolaryngology–Head and Neck Surgery, online erschienen am 19.03.2025. Verfasst von Eleonora M C Trecca und 10 weiteren Autor:innen. Beteiligte Einrichtung: GOS, Young Otolaryngologists Group of the Italian Society of Otorhinolaryngology-Head and Neck Surgery, Rome, Italy. Hauptthemen laut MEDLINE: Otolaryngology, Pediatrics, Internship and Residency, Artificial Intelligence. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 20 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

Pediatric otorhinolaryngology (ORL) addresses complex conditions in children, requiring a tailored approach for patients and families. With artificial intelligence (AI) gaining traction in medical applications, this study evaluates the quality of information provided by large language models (LLMs) in comparison to clinicians, identifying strengths and limitations in the field of pediatric ORL. Study Design Comparative blinded study.

Setting

Controlled research environment using LLMs.

Methods

Fifty‐four items of increasing difficulty, namely 18 theoretical questions, 18 clinical scenarios, and 18 patient questions, were posed to ChatGPT‐3.5, ‐4.0, ‐4o, Claude‐3, Gemini, Perplexity, Copilot, a second‐year resident, and an expert in the field of pediatric ORL. The Quality Analysis of Medical Artificial Intelligence (QAMAI) tool was used for blinded evaluation of the quality of medical information by a panel of expert members from the Young Otolaryngologists Group of the Italian Society of ORL and the International Federation of ORL Societies.

Results

LLMs performed comparably to specialist in theoretical and standardized clinical scenarios, with Bing Copilot achieving the highest QAMAI scores. However, AI responses lacked transparency in citing reliable sources and were less effective in addressing patient‐centered questions. Poor interrater agreement among reviewers highlighted challenges in distinguishing human‐generated from AI‐generated responses. Rhinology topics received the highest scores, whereas laryngology and patient‐centered questions showed lower agreement and performance.

Conclusion

LLMs show promise as supportive resources in pediatric ORL, particularly in theoretical learning and standardized cases. However, significant limitations remain, including source transparency and contextual communication in patient interactions. Human oversight is essential to mitigate risks. Future developments should focus on refining AI capabilities for evidence‐based and empathetic communication to support both clinicians and families.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

OtolaryngologyPediatricsInternship and ResidencyArtificial IntelligenceLanguageData AccuracyHumansChildLarge Language Models

Stichwörter der Autor:innen

ChatGPTartificial intelligencedigital healtheHealthlarge language modelsotolaryngologypediatric otorhinolaryngologypediatrics

Autor:innen und Institutionen

  • GOS, Young Otolaryngologists Group of the Italian Society of Otorhinolaryngology-Head and Neck Surgery, Rome, Italy.
  • Department of Otorhinolaryngology and Maxillofacial Surgery, IRCCS Research Hospital Casa Sollievo della Sofferenza, San Giovanni Rotondo, Foggia, Italy.
  • Department of Clinical and Experimental Medicine, University of Foggia, Foggia, Italy.
  • Department of Otolaryngology-Head and Neck Surgery, ASST Lariana, Ospedale Sant'Anna, University of Insubria, Como, Italy.
  • Department of Biotechnology and LifeSciences, University of Insubria, Varese, Italy.
  • Otorhinolaryngology-Head and Neck Surgery Unit, AORN "San Pio", Benevento, Italy.

ORCID-Profile: Antonella Miriam di Lullo, Giuseppe Maruccio

Zitieren

APA

Trecca, E. M. C., Caponio, V. C. A., Turri-Zanoni, M., di Lullo, A. M., Gaffuri, M., Lechien, J. R., Maniaci, A., Maruccio, G., Reale, M., Visconti, I. C., & Dallari, V. (2025). Comparative Analysis of Information Quality in Pediatric Otorhinolaryngology: Clinicians, Residents, and Large Language Models. Otolaryngology–Head and Neck Surgery, 173(1), 228-236. https://doi.org/10.1002/ohn.1225

Vancouver

Trecca EMC, Caponio VCA, Turri-Zanoni M, di Lullo AM, Gaffuri M, Lechien JR, et al. Comparative Analysis of Information Quality in Pediatric Otorhinolaryngology: Clinicians, Residents, and Large Language Models. Otolaryngology–Head and Neck Surgery. 2025;173(1):228-236. doi:10.1002/ohn.1225

BibTeX
@article{Trecca2025,
  author = {Trecca, Eleonora M C and Caponio, Vito Carlo Alberto and Turri-Zanoni, Mario and di Lullo, Antonella Miriam and Gaffuri, Michele and Lechien, Jérôme R and Maniaci, Antonino and Maruccio, Giuseppe and Reale, Marella and Visconti, Irene Claudia and Dallari, Virginia},
  title = {Comparative Analysis of Information Quality in Pediatric Otorhinolaryngology: Clinicians, Residents, and Large Language Models},
  journal = {Otolaryngology–Head and Neck Surgery},
  year = {2025},
  volume = {173},
  number = {1},
  pages = {228--236},
  doi = {10.1002/ohn.1225}
}
Kostenlos starten →