Evaluating DeepSeek in Real-World Clinical Scenarios on Performance and Safeguards: Multicenter Comparative Study
Journal of Medical Internet Research · 2026-11-13 · 28 · e86746-e86746
DOI: 10.2196/86746 ↗At a glance
Multizentrische Studie in Journal of Medical Internet Research, published online on 2026-11-13. Written by Zuyi Yang and 19 co-authors. Institution: Department of Ophthalmology, Key Lab of Ocular Fundus Diseases, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking…. Major MEDLINE topics: Electronic Health Records, Generative Artificial Intelligence. 38 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
DeepSeek, an open-source large language model (LLM) extensively adopted in hospitals across China, presents a locally deployable alternative to proprietary models such as GPT. However, its real-world clinical performance and safeguards remain underexplored.
Objective
The aim of this study is to evaluate the performance of the general-purpose and reasoning versions of DeepSeek (DeepSeek AI) and GPT models in diagnosis, medical record analysis, and safeguards against adversarial vulnerabilities.
Methods
This cross-sectional study included 50 real-world medical records from 10 hospitals spanning 14 specialties. A total of 7 LLMs were evaluated, including DeepSeek-V3, DeepSeek-R1, GPT-4o (via ChatGPT; OpenAI), o3-mini (via ChatGPT), o3-mini-high (via ChatGPT), GPT-4o (via Copilot; Microsoft), and GPT-4o-reasoning (via Copilot). Models were tasked with generating diagnoses based on admission notes, answering 12 questions related to extraction, summarization, and inference using discharge summaries, and undergoing 4 rounds of simulated adversarial prompt attacks designed to elicit misinformation. The diagnostic evaluation was performed on 47 records after the exclusion of 3 flawed admission notes, while the other tasks used the full set of 50 records. Diagnostic performance was evaluated based on accuracy, with the causes of misdiagnosis further categorized. Medical record analysis was assessed by scores in extraction, summarization, and inference tasks. Safeguard evaluation included identifying the earliest round of successful prompt attack and verifying reference authenticity and accuracy.
Results
Among the 7 models included in this study, DeepSeek-R1 and GPT-4o-reasoning (via Copilot) achieved the highest diagnostic accuracy at 64% (30/47). In medical record analysis, both models also excelled across the 3 subdimensions. GPT-4o-reasoning (via Copilot) achieved accuracies of 97.7% (293/300) in extraction, 87.5% (175/200) in summarization, and 87% in inference, while DeepSeek-R1 reached 97.7% (293/300), 83.5% (167/200), and 85% (85/100), respectively. In simulated adversarial safeguard evaluation involving 4 rounds of prompt attacks, all evaluated models showed high vulnerability. DeepSeek-R1 generated medical disinformation in 98% (49/50) of cases, with only a 2% refusal rate. GPT-4o-reasoning (via Copilot) showed the highest refusal rate at 30%, but it still generated disinformation in 70% (35/50) of cases. True hallucination rates ranged from 38.6% (27/70; GPT-4o-reasoning via Copilot) to 73% (73/100; DeepSeek-V3).
Conclusions
These findings underscore the clinical application potential of DeepSeek models, while also emphasizing the critical need to strengthen safeguards against adversarial vulnerabilities prior to their clinical deployment.
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Department of Ophthalmology, Key Lab of Ocular Fundus Diseases, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, No.1 Shuaifuyuan, Wangfujing, Dongcheng Distri…
- Department of Liver Surgery, State Key Laboratory of Complex Severe and Rare Diseases, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, No.1 Shuaifuyuan, Wangf…
- School of Medicine, Southeast University, Naijing, China.
- Department of Ophthalmology, Pingdingshan First People's Hospital, Pingdingshan, China.
- Department of Ophthalmology, The First Affiliated Hospital of Soochow University, Suzhou, China.
- Department of Ophthalmology, Tianjin Medical University General Hospital, Tianjin, China.
Cite
APA
Yang, Z., Tian, D., He, M., Wang, E., Zhao, L., Zhao, X., Chen, X., Xia, W., Zhou, W., Ma, Y., Ma, C., Chen, H., Lu, H., Liu, S., Zhao, X., Xu, Y., Lu, X., Wang, F., Zhang, L., & Chen, Y. (2026). Evaluating DeepSeek in Real-World Clinical Scenarios on Performance and Safeguards: Multicenter Comparative Study. Journal of Medical Internet Research, 28, e86746-e86746. https://doi.org/10.2196/86746
Vancouver
Yang Z, Tian D, He M, Wang E, Zhao L, Zhao X, et al. Evaluating DeepSeek in Real-World Clinical Scenarios on Performance and Safeguards: Multicenter Comparative Study. Journal of Medical Internet Research. 2026;28:e86746-e86746. doi:10.2196/86746
BibTeX
@article{Yang2026,
author = {Yang, Zuyi and Tian, Dianzhe and He, Meiqian and Wang, Erqian and Zhao, Lvyuxing and Zhao, Xinyu and Chen, Xiaokai and Xia, Wei and Zhou, Wei and Ma, Yanhao and Ma, Chuanyong and Chen, Haiyan and Lu, Huiqin and Liu, Shulin and Zhao, Xin and Xu, Yiyao and Lu, Xin and Wang, Fengdan and Zhang, Lei and Chen, Youxin},
title = {Evaluating DeepSeek in Real-World Clinical Scenarios on Performance and Safeguards: Multicenter Comparative Study},
journal = {Journal of Medical Internet Research},
year = {2026},
volume = {28},
pages = {e86746--e86746},
doi = {10.2196/86746}
}
EUVIMED