EUVIMEDOpdag EUVIMED
VergleichsstudieQuerschnittstudieOpen access

Evaluating the Performance of DeepSeek-R1 and DeepSeek-V3 Versus OpenAI Models in the Chinese National Medical Licensing Examination: Cross-Sectional Comparative Study

Weiping Wang iD, Yuchen Zhou iD, Jingxuan Fu iD, Ke Hu iD

JMIR Medical Education · 2025-11-14 · 11 · e73469

DOI: 10.2196/73469 ↗

At a glance

Vergleichsstudie in JMIR Medical Education, published online on 2025-11-14. Written by Weiping Wang and 3 co-authors. Institution: Department of Radiation Oncology, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences & Peking Union Medical College, Beijing,…. Major MEDLINE topics: Licensure, Medical, Educational Measurement, Language. Cited 22 times so far (Crossref); 22 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

Deepseek-R1, an open-source large language model (LLM), has generated significant global interest in the past months.

Objective

This study aimed to compare the performance of DeepSeek and OpenAI LLMs on the Chinese National Medical Licensing Examination (NMLE) and evaluate their potential in medical education.

Methods

This cross-sectional study assessed 2 DeepSeek models (DeepSeek-R1 and DeepSeek-V3), 3 OpenAI models (ChatGPT-o1 pro, ChatGPT-o3 mini, and GPT-4o), and 2 additional Chinese LLMs (ERNIE 4.5 Turbo and Qwen 3) using the 2021 NMLE. Model performance was evaluated based on overall accuracy, accuracy across question types (A1, A2, A3 and A4, and B1), case analysis and non–case analysis questions, medical specialties, and accuracy consensus between different model combinations.

Results

All LLMs successfully passed the NMLE. DeepSeek-R1 achieved the highest accuracy (573/597, 96%), followed by DeepSeek-V3 (558/600, 93%), both of which significantly outperformed ChatGPT-o1 pro (450/600, 75%), ChatGPT-o3 mini (455/600, 75.8%), and GPT-4o (452/600, 75.3%; P

Subjects

MeSH terms (MEDLINE)

Licensure, MedicalEducational MeasurementLanguageCross-Sectional StudiesHumansChina

Author keywords

AIChinese Medical Licensing ExaminationDeepSeekOpenAIartificial intelligencelarge language models

Authors and institutions

  • Department of Radiation Oncology, Peking Union Medical College Hospital, Chinese Academy of Medical Sciences & Peking Union Medical College, Beijing, China.
  • Tsinghua Medicine, School of Medicine, Tsinghua University, Beijing, China.
  • Department of clinical laboratory, Xuanwu Hospital, Capital Medical University, Beijing, China.

ORCID profiles: Weiping Wang, Yuchen Zhou, Jingxuan Fu, Ke Hu

Cite

APA

Wang, W., Zhou, Y., Fu, J., & Hu, K. (2025). Evaluating the Performance of DeepSeek-R1 and DeepSeek-V3 Versus OpenAI Models in the Chinese National Medical Licensing Examination: Cross-Sectional Comparative Study. JMIR Medical Education, 11, e73469. https://doi.org/10.2196/73469

Vancouver

Wang W, Zhou Y, Fu J, Hu K. Evaluating the Performance of DeepSeek-R1 and DeepSeek-V3 Versus OpenAI Models in the Chinese National Medical Licensing Examination: Cross-Sectional Comparative Study. JMIR Medical Education. 2025;11:e73469. doi:10.2196/73469

BibTeX
@article{Wang2025,
  author = {Wang, Weiping and Zhou, Yuchen and Fu, Jingxuan and Hu, Ke},
  title = {Evaluating the Performance of DeepSeek-R1 and DeepSeek-V3 Versus OpenAI Models in the Chinese National Medical Licensing Examination: Cross-Sectional Comparative Study},
  journal = {JMIR Medical Education},
  year = {2025},
  volume = {11},
  pages = {e73469},
  doi = {10.2196/73469}
}
Start for free →