EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

A clinician-based comparative study of large language models in answering medical questions: the case of asthma

Yong Yin, Mei Zeng, Hansong Wang, Haibo Yang, Caijing Zhou, Feng Jiang, Shufan Wu, Tingyue Huang, Shuahua Yuan, Jilei Lin, Mingyu Tang, Jiande Chen und 3 weitere

Frontiers in Pediatrics · 25.04.2025 · 13

DOI: 10.3389/fped.2025.1461026 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Frontiers in Pediatrics, online erschienen am 25.04.2025. Verfasst von Yong Yin und 14 weiteren Autor:innen. Beteiligte Einrichtung: Department of Respiratory Medicine, Hainan Branch, Shanghai Children's Medical Center, School of Medicine, Shanghai Jiao Tong University, Sanya, Chin…. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 19 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

This study aims to evaluate and compare the performance of four major large language models (GPT-3.5, GPT-4.0, YouChat, and Perplexity) in answering 32 common asthma-related questions. Materials and methods Seventy-five clinicians from various tertiary hospitals participated in this study. Each clinician was tasked with evaluating the responses generated by the four large language models (LLMs) to 32 common clinical questions related to pediatric asthma. Based on predefined criteria, participants subjectively assessed the accuracy, correctness, completeness, and practicality of the LLMs' answers. The participants provided precise scores to determine the performance of each language model in answering pediatric asthma-related questions.

Results

GPT-4.0 performed the best across all dimensions, while YouChat performed the worst in all dimensions. Both GPT-3.5 and GPT-4.0 outperformed the other two models, but there was no significant difference in performance between GPT-3.5 and GPT-4.0 or between YouChat and Perplexity.

Conclusion

GPT and other large language models can answer medical questions with a certain degree of completeness and accuracy. However, clinical physicians should critically assess internet information, distinguishing between true and false data, and should not blindly accept the outputs of these models. With advancements in key technologies, LLMs may one day become a safe option for doctors seeking information.

Schlagwörter

Stichwörter der Autor:innen

AI in healthcareclinical decision supportlarge language models (LLMs)medical question answeringpediatric asthma

Autor:innen und Institutionen

  • Department of Respiratory Medicine, Hainan Branch, Shanghai Children's Medical Center, School of Medicine, Shanghai Jiao Tong University, Sanya, China.
  • Department of Respiratory Medicine, Shanghai Children's Medical Center, Shanghai Jiao Tong University School of Medicine, Shanghai, China.
  • Pediatric AI Clinical Application and Research Center, Shanghai Children's Medical Center, Shanghai, China.
  • Shanghai Engineering Research Center of Intelligence Pediatrics (SERCIP), Shanghai, China.
  • Department of Performance, Shanghai Children's Medical Center, Shanghai Jiao Tong University School of Medicine, Shanghai, China.
  • Department of Discipline Inspection and Supervision, Shanghai Children's Medical Center, Shanghai Jiao Tong University School of Medicine, Shanghai, China.

Zitieren

APA

Yin, Y., Zeng, M., Wang, H., Yang, H., Zhou, C., Jiang, F., Wu, S., Huang, T., Yuan, S., Lin, J., Tang, M., Chen, J., Dong, B., Yuan, J., & Xie, D. (2025). A clinician-based comparative study of large language models in answering medical questions: the case of asthma. Frontiers in Pediatrics, 13. https://doi.org/10.3389/fped.2025.1461026

Vancouver

Yin Y, Zeng M, Wang H, Yang H, Zhou C, Jiang F, et al. A clinician-based comparative study of large language models in answering medical questions: the case of asthma. Frontiers in Pediatrics. 2025;13. doi:10.3389/fped.2025.1461026

BibTeX
@article{Yin2025,
  author = {Yin, Yong and Zeng, Mei and Wang, Hansong and Yang, Haibo and Zhou, Caijing and Jiang, Feng and Wu, Shufan and Huang, Tingyue and Yuan, Shuahua and Lin, Jilei and Tang, Mingyu and Chen, Jiande and Dong, Bin and Yuan, Jiajun and Xie, Dan},
  title = {A clinician-based comparative study of large language models in answering medical questions: the case of asthma},
  journal = {Frontiers in Pediatrics},
  year = {2025},
  volume = {13},
  doi = {10.3389/fped.2025.1461026}
}
Kostenlos starten →