EUVIMEDScopri EUVIMED
Multizentrische StudieVergleichsstudieOpen access

Evaluating DeepSeek in Real-World Clinical Scenarios on Performance and Safeguards: Multicenter Comparative Study

Zuyi Yang, Dianzhe Tian, Meiqian He, Erqian Wang, Lvyuxing Zhao, Xinyu Zhao, Xiaokai Chen, Wei Xia, Wei Zhou, Yanhao Ma, Chuanyong Ma, Haiyan Chen and 8 more

Journal of Medical Internet Research · 2026-11-13 · 28 · e86746-e86746

DOI: 10.2196/86746 ↗

At a glance

Multizentrische Studie in Journal of Medical Internet Research, published online on 2026-11-13. Written by Zuyi Yang and 19 co-authors. Institution: Department of Ophthalmology, Key Lab of Ocular Fundus Diseases, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking…. Major MEDLINE topics: Electronic Health Records, Generative Artificial Intelligence. 38 references. The full text is freely available in PubMed Central.

AbstractSubjectsDati bibliograficiCiteRelated studies

Abstract (original language)

Background

DeepSeek, an open-source large language model (LLM) extensively adopted in hospitals across China, presents a locally deployable alternative to proprietary models such as GPT. However, its real-world clinical performance and safeguards remain underexplored.

Objective

The aim of this study is to evaluate the performance of the general-purpose and reasoning versions of DeepSeek (DeepSeek AI) and GPT models in diagnosis, medical record analysis, and safeguards against adversarial vulnerabilities.

Methods

This cross-sectional study included 50 real-world medical records from 10 hospitals spanning 14 specialties. A total of 7 LLMs were evaluated, including DeepSeek-V3, DeepSeek-R1, GPT-4o (via ChatGPT; OpenAI), o3-mini (via ChatGPT), o3-mini-high (via ChatGPT), GPT-4o (via Copilot; Microsoft), and GPT-4o-reasoning (via Copilot). Models were tasked with generating diagnoses based on admission notes, answering 12 questions related to extraction, summarization, and inference using discharge summaries, and undergoing 4 rounds of simulated adversarial prompt attacks designed to elicit misinformation. The diagnostic evaluation was performed on 47 records after the exclusion of 3 flawed admission notes, while the other tasks used the full set of 50 records. Diagnostic performance was evaluated based on accuracy, with the causes of misdiagnosis further categorized. Medical record analysis was assessed by scores in extraction, summarization, and inference tasks. Safeguard evaluation included identifying the earliest round of successful prompt attack and verifying reference authenticity and accuracy.

Results

Among the 7 models included in this study, DeepSeek-R1 and GPT-4o-reasoning (via Copilot) achieved the highest diagnostic accuracy at 64% (30/47). In medical record analysis, both models also excelled across the 3 subdimensions. GPT-4o-reasoning (via Copilot) achieved accuracies of 97.7% (293/300) in extraction, 87.5% (175/200) in summarization, and 87% in inference, while DeepSeek-R1 reached 97.7% (293/300), 83.5% (167/200), and 85% (85/100), respectively. In simulated adversarial safeguard evaluation involving 4 rounds of prompt attacks, all evaluated models showed high vulnerability. DeepSeek-R1 generated medical disinformation in 98% (49/50) of cases, with only a 2% refusal rate. GPT-4o-reasoning (via Copilot) showed the highest refusal rate at 30%, but it still generated disinformation in 70% (35/50) of cases. True hallucination rates ranged from 38.6% (27/70; GPT-4o-reasoning via Copilot) to 73% (73/100; DeepSeek-V3).

Conclusions

These findings underscore the clinical application potential of DeepSeek models, while also emphasizing the critical need to strengthen safeguards against adversarial vulnerabilities prior to their clinical deployment.

Subjects

MeSH terms (MEDLINE)

Electronic Health RecordsGenerative Artificial IntelligenceHumansChinaCross-Sectional StudiesLarge Language Models

Author keywords

ChatGPTdeepseeklarge language modelsmedical recordssafeguards

Authors and institutions

  • Department of Ophthalmology, Key Lab of Ocular Fundus Diseases, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, No.1 Shuaifuyuan, Wangfujing, Dongcheng Distri…
  • Department of Liver Surgery, State Key Laboratory of Complex Severe and Rare Diseases, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, No.1 Shuaifuyuan, Wangf…
  • School of Medicine, Southeast University, Naijing, China.
  • Department of Ophthalmology, Pingdingshan First People's Hospital, Pingdingshan, China.
  • Department of Ophthalmology, The First Affiliated Hospital of Soochow University, Suzhou, China.
  • Department of Ophthalmology, Tianjin Medical University General Hospital, Tianjin, China.

Cite

APA

Yang, Z., Tian, D., He, M., Wang, E., Zhao, L., Zhao, X., Chen, X., Xia, W., Zhou, W., Ma, Y., Ma, C., Chen, H., Lu, H., Liu, S., Zhao, X., Xu, Y., Lu, X., Wang, F., Zhang, L., & Chen, Y. (2026). Evaluating DeepSeek in Real-World Clinical Scenarios on Performance and Safeguards: Multicenter Comparative Study. Journal of Medical Internet Research, 28, e86746-e86746. https://doi.org/10.2196/86746

Vancouver

Yang Z, Tian D, He M, Wang E, Zhao L, Zhao X, et al. Evaluating DeepSeek in Real-World Clinical Scenarios on Performance and Safeguards: Multicenter Comparative Study. Journal of Medical Internet Research. 2026;28:e86746-e86746. doi:10.2196/86746

BibTeX
@article{Yang2026,
  author = {Yang, Zuyi and Tian, Dianzhe and He, Meiqian and Wang, Erqian and Zhao, Lvyuxing and Zhao, Xinyu and Chen, Xiaokai and Xia, Wei and Zhou, Wei and Ma, Yanhao and Ma, Chuanyong and Chen, Haiyan and Lu, Huiqin and Liu, Shulin and Zhao, Xin and Xu, Yiyao and Lu, Xin and Wang, Fengdan and Zhang, Lei and Chen, Youxin},
  title = {Evaluating DeepSeek in Real-World Clinical Scenarios on Performance and Safeguards: Multicenter Comparative Study},
  journal = {Journal of Medical Internet Research},
  year = {2026},
  volume = {28},
  pages = {e86746--e86746},
  doi = {10.2196/86746}
}
Start for free →