Comparative Analysis of Information Quality in Pediatric Otorhinolaryngology: Clinicians, Residents, and Large Language Models
Otolaryngology–Head and Neck Surgery · 2025-07 · 173 (1) · 228-236
DOI: 10.1002/ohn.1225 ↗At a glance
Vergleichsstudie in Otolaryngology–Head and Neck Surgery, published online on 2025-03-19. Written by Eleonora M C Trecca and 10 co-authors. Institution: GOS, Young Otolaryngologists Group of the Italian Society of Otorhinolaryngology-Head and Neck Surgery, Rome, Italy. Major MEDLINE topics: Otolaryngology, Pediatrics, Internship and Residency, Artificial Intelligence. Cited 15 times so far (Crossref); 20 references. The full text is freely available (open access, CC BY 4.0).
Abstract (original language)
Objective
Pediatric otorhinolaryngology (ORL) addresses complex conditions in children, requiring a tailored approach for patients and families. With artificial intelligence (AI) gaining traction in medical applications, this study evaluates the quality of information provided by large language models (LLMs) in comparison to clinicians, identifying strengths and limitations in the field of pediatric ORL. Study Design Comparative blinded study.
Setting
Controlled research environment using LLMs.
Methods
Fifty‐four items of increasing difficulty, namely 18 theoretical questions, 18 clinical scenarios, and 18 patient questions, were posed to ChatGPT‐3.5, ‐4.0, ‐4o, Claude‐3, Gemini, Perplexity, Copilot, a second‐year resident, and an expert in the field of pediatric ORL. The Quality Analysis of Medical Artificial Intelligence (QAMAI) tool was used for blinded evaluation of the quality of medical information by a panel of expert members from the Young Otolaryngologists Group of the Italian Society of ORL and the International Federation of ORL Societies.
Results
LLMs performed comparably to specialist in theoretical and standardized clinical scenarios, with Bing Copilot achieving the highest QAMAI scores. However, AI responses lacked transparency in citing reliable sources and were less effective in addressing patient‐centered questions. Poor interrater agreement among reviewers highlighted challenges in distinguishing human‐generated from AI‐generated responses. Rhinology topics received the highest scores, whereas laryngology and patient‐centered questions showed lower agreement and performance.
Conclusion
LLMs show promise as supportive resources in pediatric ORL, particularly in theoretical learning and standardized cases. However, significant limitations remain, including source transparency and contextual communication in patient interactions. Human oversight is essential to mitigate risks. Future developments should focus on refining AI capabilities for evidence‐based and empathetic communication to support both clinicians and families.
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- GOS, Young Otolaryngologists Group of the Italian Society of Otorhinolaryngology-Head and Neck Surgery, Rome, Italy.
- Department of Otorhinolaryngology and Maxillofacial Surgery, IRCCS Research Hospital Casa Sollievo della Sofferenza, San Giovanni Rotondo, Foggia, Italy.
- Department of Clinical and Experimental Medicine, University of Foggia, Foggia, Italy.
- Department of Otolaryngology-Head and Neck Surgery, ASST Lariana, Ospedale Sant'Anna, University of Insubria, Como, Italy.
- Department of Biotechnology and LifeSciences, University of Insubria, Varese, Italy.
- Otorhinolaryngology-Head and Neck Surgery Unit, AORN "San Pio", Benevento, Italy.
ORCID profiles: Antonella Miriam di Lullo, Giuseppe Maruccio
Cite
APA
Trecca, E. M. C., Caponio, V. C. A., Turri-Zanoni, M., di Lullo, A. M., Gaffuri, M., Lechien, J. R., Maniaci, A., Maruccio, G., Reale, M., Visconti, I. C., & Dallari, V. (2025). Comparative Analysis of Information Quality in Pediatric Otorhinolaryngology: Clinicians, Residents, and Large Language Models. Otolaryngology–Head and Neck Surgery, 173(1), 228-236. https://doi.org/10.1002/ohn.1225
Vancouver
Trecca EMC, Caponio VCA, Turri-Zanoni M, di Lullo AM, Gaffuri M, Lechien JR, et al. Comparative Analysis of Information Quality in Pediatric Otorhinolaryngology: Clinicians, Residents, and Large Language Models. Otolaryngology–Head and Neck Surgery. 2025;173(1):228-236. doi:10.1002/ohn.1225
BibTeX
@article{Trecca2025,
author = {Trecca, Eleonora M C and Caponio, Vito Carlo Alberto and Turri-Zanoni, Mario and di Lullo, Antonella Miriam and Gaffuri, Michele and Lechien, Jérôme R and Maniaci, Antonino and Maruccio, Giuseppe and Reale, Marella and Visconti, Irene Claudia and Dallari, Virginia},
title = {Comparative Analysis of Information Quality in Pediatric Otorhinolaryngology: Clinicians, Residents, and Large Language Models},
journal = {Otolaryngology–Head and Neck Surgery},
year = {2025},
volume = {173},
number = {1},
pages = {228--236},
doi = {10.1002/ohn.1225}
}
EUVIMED