A clinician-based comparative study of large language models in answering medical questions: the case of asthma
Frontiers in Pediatrics · 25.04.2025 · 13
DOI: 10.3389/fped.2025.1461026 ↗Kurz gefasst
Fachartikel in der Zeitschrift Frontiers in Pediatrics, online erschienen am 25.04.2025. Verfasst von Yong Yin und 14 weiteren Autor:innen. Beteiligte Einrichtung: Department of Respiratory Medicine, Hainan Branch, Shanghai Children's Medical Center, School of Medicine, Shanghai Jiao Tong University, Sanya, Chin…. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 19 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Objective
This study aims to evaluate and compare the performance of four major large language models (GPT-3.5, GPT-4.0, YouChat, and Perplexity) in answering 32 common asthma-related questions. Materials and methods Seventy-five clinicians from various tertiary hospitals participated in this study. Each clinician was tasked with evaluating the responses generated by the four large language models (LLMs) to 32 common clinical questions related to pediatric asthma. Based on predefined criteria, participants subjectively assessed the accuracy, correctness, completeness, and practicality of the LLMs' answers. The participants provided precise scores to determine the performance of each language model in answering pediatric asthma-related questions.
Results
GPT-4.0 performed the best across all dimensions, while YouChat performed the worst in all dimensions. Both GPT-3.5 and GPT-4.0 outperformed the other two models, but there was no significant difference in performance between GPT-3.5 and GPT-4.0 or between YouChat and Perplexity.
Conclusion
GPT and other large language models can answer medical questions with a certain degree of completeness and accuracy. However, clinical physicians should critically assess internet information, distinguishing between true and false data, and should not blindly accept the outputs of these models. With advancements in key technologies, LLMs may one day become a safe option for doctors seeking information.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Respiratory Medicine, Hainan Branch, Shanghai Children's Medical Center, School of Medicine, Shanghai Jiao Tong University, Sanya, China.
- Department of Respiratory Medicine, Shanghai Children's Medical Center, Shanghai Jiao Tong University School of Medicine, Shanghai, China.
- Pediatric AI Clinical Application and Research Center, Shanghai Children's Medical Center, Shanghai, China.
- Shanghai Engineering Research Center of Intelligence Pediatrics (SERCIP), Shanghai, China.
- Department of Performance, Shanghai Children's Medical Center, Shanghai Jiao Tong University School of Medicine, Shanghai, China.
- Department of Discipline Inspection and Supervision, Shanghai Children's Medical Center, Shanghai Jiao Tong University School of Medicine, Shanghai, China.
Zitieren
APA
Yin, Y., Zeng, M., Wang, H., Yang, H., Zhou, C., Jiang, F., Wu, S., Huang, T., Yuan, S., Lin, J., Tang, M., Chen, J., Dong, B., Yuan, J., & Xie, D. (2025). A clinician-based comparative study of large language models in answering medical questions: the case of asthma. Frontiers in Pediatrics, 13. https://doi.org/10.3389/fped.2025.1461026
Vancouver
Yin Y, Zeng M, Wang H, Yang H, Zhou C, Jiang F, et al. A clinician-based comparative study of large language models in answering medical questions: the case of asthma. Frontiers in Pediatrics. 2025;13. doi:10.3389/fped.2025.1461026
BibTeX
@article{Yin2025,
author = {Yin, Yong and Zeng, Mei and Wang, Hansong and Yang, Haibo and Zhou, Caijing and Jiang, Feng and Wu, Shufan and Huang, Tingyue and Yuan, Shuahua and Lin, Jilei and Tang, Mingyu and Chen, Jiande and Dong, Bin and Yuan, Jiajun and Xie, Dan},
title = {A clinician-based comparative study of large language models in answering medical questions: the case of asthma},
journal = {Frontiers in Pediatrics},
year = {2025},
volume = {13},
doi = {10.3389/fped.2025.1461026}
}
EUVIMED