EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Performance evaluation of large language models in the diagnosis of emergency internal medicine diseases: a retrospective study

Jintao Wei, Shouyin Jiang, Ting Yin, Jiangchun Ma, Qiang Li, Yu Tian, Mengting Yan, Zhuyi Shen, Xiangkang Lv, Ximei Ma, Shanxiang Xu, Mao Zhang

Frontiers in Public Health · 2026

DOI: 10.3389/fpubh.2026.1780425 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Frontiers in Public Health, 2026. Verfasst von Jintao Wei und 11 weiteren Autor:innen. Beteiligte Einrichtung: Department of Emergency Medicine, Second Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China. Hauptthemen laut MEDLINE: Emergency Medicine, Large Language Models, Decision Support Systems, Clinical. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 30 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

Medical domain large language models (LLMs) exhibit verified clinical decision-support capabilities in simulated case analyses and standardized tests, yet their diagnostic efficacy in real-world emergency settings remain insufficiently explored. This study evaluates the diagnostic performance of 5 mainstream LLMs (ChatGPT-4o, Gemini-2.0, Grok3, DeepSeek-V3, Doubao) against emergency department junior physicians (EDJP) on real-world emergency internal medicine cases.

Methods

A single-center retrospective analysis design was conducted. 154 anonymized emergency internal medicine patients of the Second Affiliated Hospital of Zhejiang University School of Medicine from January to May 2025 were included, covering common acute diseases of multiple systems. 15 EDJPs and 5 LLMs were selected to diagnose the cases, respectively. The main diagnostic accuracy, comprehensiveness of differential diagnosis, and response time were used as evaluation indicators. Non-parametric tests were used for statistical analysis.

Results

(1) Main diagnostic accuracy: DeepSeek-V3 (90.0%), ChatGPT-4o (86.0%), and Grok3 (86.0%) were significantly higher than that of EDJP (77.5%, p < 0.05); in the subgroup of respiratory system diseases, Gemini-2.0 and DeepSeek-V3 performed better ( p < 0.05). (2) Comprehensiveness of differential diagnosis: The scores of all LLMs were significantly higher than that of EDJP ( p < 0.05), and the medians of DeepSeek-V3, Gemini-2.0, and Grok3 reached 5.0 points. (3) Response time: LLMs (6.3–14.0 s) were significantly faster than EDJP (360.2 s, p < 0.05), and Doubao had the fastest response. The inter-rater reliability was good (ICC: 0.617–0.899).

Conclusion

This retrospective study shows that LLMs outperformed EDJPs in diagnostic accuracy, differential diagnosis comprehensiveness and response efficiency for emergency internal medicine diseases, demonstrating significant potential for clinical decision support. Subsequent efforts will focus on exploring how to effectively integrate into physician-led collaborative workflows to enhance emergency care quality and efficiency.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Emergency MedicineLarge Language ModelsDecision Support Systems, ClinicalRetrospective StudiesHumansAdultMiddle AgedMaleFemaleDiagnosisChina

Stichwörter der Autor:innen

DeepSeek-V3GPT-4odiagnostic accuracyemergency department junior physicianslarge language model

Autor:innen und Institutionen

  • Department of Emergency Medicine, Second Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China.
  • Key Laboratory of the Diagnosis and Treatment of Severe Trauma and Burn of Zhejiang Province, Hangzhou, China.
  • Zhejiang Province Clinical Research Center for Emergency and Critical Care Medicine, Hangzhou, China.
  • National Emergency Medical Rescue Base, Hangzhou, China.
  • Department of Cardiology, The Second Affiliated Hospital, School of Medicine, Zhejiang University, Hangzhou, China.
  • Department of Neurosurgery, Zhejiang Hospital, Hangzhou, China.

Zitieren

APA

Wei, J., Jiang, S., Yin, T., Ma, J., Li, Q., Tian, Y., Yan, M., Shen, Z., Lv, X., Ma, X., Xu, S., & Zhang, M. (2026). Performance evaluation of large language models in the diagnosis of emergency internal medicine diseases: a retrospective study. Frontiers in Public Health. https://doi.org/10.3389/fpubh.2026.1780425

Vancouver

Wei J, Jiang S, Yin T, Ma J, Li Q, Tian Y, et al. Performance evaluation of large language models in the diagnosis of emergency internal medicine diseases: a retrospective study. Frontiers in Public Health. 2026. doi:10.3389/fpubh.2026.1780425

BibTeX
@article{Wei2026,
  author = {Wei, Jintao and Jiang, Shouyin and Yin, Ting and Ma, Jiangchun and Li, Qiang and Tian, Yu and Yan, Mengting and Shen, Zhuyi and Lv, Xiangkang and Ma, Ximei and Xu, Shanxiang and Zhang, Mao},
  title = {Performance evaluation of large language models in the diagnosis of emergency internal medicine diseases: a retrospective study},
  journal = {Frontiers in Public Health},
  year = {2026},
  doi = {10.3389/fpubh.2026.1780425}
}
Kostenlos starten →