Performance evaluation of large language models in the diagnosis of emergency internal medicine diseases: a retrospective study
Frontiers in Public Health · 2026
DOI: 10.3389/fpubh.2026.1780425 ↗Kurz gefasst
Fachartikel in der Zeitschrift Frontiers in Public Health, 2026. Verfasst von Jintao Wei und 11 weiteren Autor:innen. Beteiligte Einrichtung: Department of Emergency Medicine, Second Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China. Hauptthemen laut MEDLINE: Emergency Medicine, Large Language Models, Decision Support Systems, Clinical. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 30 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Objective
Medical domain large language models (LLMs) exhibit verified clinical decision-support capabilities in simulated case analyses and standardized tests, yet their diagnostic efficacy in real-world emergency settings remain insufficiently explored. This study evaluates the diagnostic performance of 5 mainstream LLMs (ChatGPT-4o, Gemini-2.0, Grok3, DeepSeek-V3, Doubao) against emergency department junior physicians (EDJP) on real-world emergency internal medicine cases.
Methods
A single-center retrospective analysis design was conducted. 154 anonymized emergency internal medicine patients of the Second Affiliated Hospital of Zhejiang University School of Medicine from January to May 2025 were included, covering common acute diseases of multiple systems. 15 EDJPs and 5 LLMs were selected to diagnose the cases, respectively. The main diagnostic accuracy, comprehensiveness of differential diagnosis, and response time were used as evaluation indicators. Non-parametric tests were used for statistical analysis.
Results
(1) Main diagnostic accuracy: DeepSeek-V3 (90.0%), ChatGPT-4o (86.0%), and Grok3 (86.0%) were significantly higher than that of EDJP (77.5%, p < 0.05); in the subgroup of respiratory system diseases, Gemini-2.0 and DeepSeek-V3 performed better ( p < 0.05). (2) Comprehensiveness of differential diagnosis: The scores of all LLMs were significantly higher than that of EDJP ( p < 0.05), and the medians of DeepSeek-V3, Gemini-2.0, and Grok3 reached 5.0 points. (3) Response time: LLMs (6.3–14.0 s) were significantly faster than EDJP (360.2 s, p < 0.05), and Doubao had the fastest response. The inter-rater reliability was good (ICC: 0.617–0.899).
Conclusion
This retrospective study shows that LLMs outperformed EDJPs in diagnostic accuracy, differential diagnosis comprehensiveness and response efficiency for emergency internal medicine diseases, demonstrating significant potential for clinical decision support. Subsequent efforts will focus on exploring how to effectively integrate into physician-led collaborative workflows to enhance emergency care quality and efficiency.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Emergency Medicine, Second Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China.
- Key Laboratory of the Diagnosis and Treatment of Severe Trauma and Burn of Zhejiang Province, Hangzhou, China.
- Zhejiang Province Clinical Research Center for Emergency and Critical Care Medicine, Hangzhou, China.
- National Emergency Medical Rescue Base, Hangzhou, China.
- Department of Cardiology, The Second Affiliated Hospital, School of Medicine, Zhejiang University, Hangzhou, China.
- Department of Neurosurgery, Zhejiang Hospital, Hangzhou, China.
Zitieren
APA
Wei, J., Jiang, S., Yin, T., Ma, J., Li, Q., Tian, Y., Yan, M., Shen, Z., Lv, X., Ma, X., Xu, S., & Zhang, M. (2026). Performance evaluation of large language models in the diagnosis of emergency internal medicine diseases: a retrospective study. Frontiers in Public Health. https://doi.org/10.3389/fpubh.2026.1780425
Vancouver
Wei J, Jiang S, Yin T, Ma J, Li Q, Tian Y, et al. Performance evaluation of large language models in the diagnosis of emergency internal medicine diseases: a retrospective study. Frontiers in Public Health. 2026. doi:10.3389/fpubh.2026.1780425
BibTeX
@article{Wei2026,
author = {Wei, Jintao and Jiang, Shouyin and Yin, Ting and Ma, Jiangchun and Li, Qiang and Tian, Yu and Yan, Mengting and Shen, Zhuyi and Lv, Xiangkang and Ma, Ximei and Xu, Shanxiang and Zhang, Mao},
title = {Performance evaluation of large language models in the diagnosis of emergency internal medicine diseases: a retrospective study},
journal = {Frontiers in Public Health},
year = {2026},
doi = {10.3389/fpubh.2026.1780425}
}
EUVIMED