EUVIMEDObjevte EUVIMED
VergleichsstudieOpen access

Multiple Large Language Models’ Performance on the Chinese Medical Licensing Examination: Quantitative Comparative Study

Yanyu Diao iD, Mengyuan Wu iD, Jingwen Xu iD, Yifeng Pan iD

JMIR Human Factors · 2025-12-16 · 12 · e77978-e77978

DOI: 10.2196/77978 ↗

At a glance

Vergleichsstudie in JMIR Human Factors, published online on 2025-12-16. Written by Yanyu Diao and 3 co-authors. Institution: The School of Big Data and Artificial Intelligence, Anhui Xinhua University, 555 Wangjiang West Road, Hefei, 230088, China, 86 15905667742. Major MEDLINE topics: Language, Educational Measurement, Licensure, Medical, Education, Medical. Cited 8 times so far (Crossref); 19 references. The full text is freely available in PubMed Central.

AbstraktSubjectsBibliografické údajeCiteRelated studies

Abstract (original language)

Background

ChatGPT excels in natural language tasks, but its performance in the Chinese National Medical Licensing Examination (NMLE) and Chinese medical education remains underexplored. Meanwhile, Chinese corpus–based large language models (LLMs) such as ERNIE Bot, Tongyi Qianwen, Doubao, and DeepSeek have emerged, yet their effectiveness in the NMLE awaits systematic evaluation.

Objective

This study aimed to quantitatively compare the performance of 6 LLMs (GPT-3.5, GPT-4, ERNIE Bot, Tongyi Qianwen, Doubao, and DeepSeek) in answering NMLE questions from 2018 to 2024 and analyze their feasibility as supplementary tools in Chinese medical education.

Methods

We selected questions from the 4 content units of the NMLE’s General Written test (2018‐2024), preprocessed image- and table-based content into standardized text, and input the questions into each model. We evaluated the accuracy, comprehensiveness, and logical coherence of the responses, with quantitative comparison centered on scores and accuracy rates against the official answer keys (passing score: 360/600).

Results

GPT-4 outperformed GPT-3.5 across all units, achieving average accuracies of 66.57% (SD 3.21%; unit 1), 69.05% (SD 2.87%; unit 2), 71.71% (SD 2.53%; unit 3), and 80.67% (SD 2.19%; unit 4), with consistent scores above the passing threshold. Among the Chinese models, DeepSeek demonstrated the highest overall performance, with an average score of 454.8 (SD 17.3) and average accuracies of 73.2% (unit 1, SD 2.89%) and 71.5% (unit 3, SD 2.64%), as well as average accuracies of 70.3% (unit 2, SD 3.02%) and 78.2% (unit 4, SD 2.47%). ERNIE Bot (mean score 442.3, SD 19.6; unit 1 accuracy =70.8%, SD 3.01%; unit 2 accuracy =68.7%, SD 3.15%; unit 3 accuracy =69.1%, SD 2.93%; unit 4 accuracy =68.3%, SD 2.76%), Tongyi Qianwen (mean score 426.5, SD 21.4; unit 1 accuracy =67.4%, SD 3.22%; unit 2 accuracy =65.9%, SD 3.31%; unit 3 accuracy =66.2%, SD 3.08%; unit 4 accuracy =67.2%, SD 2.89%), and Doubao (mean score 413.7, SD 23.1; unit 1 accuracy =65.2%, SD 3.45%; unit 2 accuracy =63.8%, SD 3.52%; unit 3 accuracy =64.1%, SD 3.27%; unit 4 accuracy =62.8%, SD 3.11%) all exceeded the passing score. DeepSeek’s overall average accuracy (75.8%, SD 2.73%) was significantly higher than those of the other Chinese models (χ²₁=11.4, P=.001 vs ERNIE Bot; χ²₁=28.7, P

Subjects

MeSH terms (MEDLINE)

LanguageEducational MeasurementLicensure, MedicalEducation, MedicalHumansChinaLarge Language Models

Author keywords

AIChatGPTChinese National Medical Licensing ExaminationERNIE BotTongyi Qianwenartificial intelligencemedical student

Authors and institutions

  • The School of Big Data and Artificial Intelligence, Anhui Xinhua University, 555 Wangjiang West Road, Hefei, 230088, China, 86 15905667742.

ORCID profiles: Yanyu Diao, Mengyuan Wu, Jingwen Xu, Yifeng Pan

Cite

APA

Diao, Y., Wu, M., Xu, J., & Pan, Y. (2025). Multiple Large Language Models’ Performance on the Chinese Medical Licensing Examination: Quantitative Comparative Study. JMIR Human Factors, 12, e77978-e77978. https://doi.org/10.2196/77978

Vancouver

Diao Y, Wu M, Xu J, Pan Y. Multiple Large Language Models’ Performance on the Chinese Medical Licensing Examination: Quantitative Comparative Study. JMIR Human Factors. 2025;12:e77978-e77978. doi:10.2196/77978

BibTeX
@article{Diao2025,
  author = {Diao, Yanyu and Wu, Mengyuan and Xu, Jingwen and Pan, Yifeng},
  title = {Multiple Large Language Models’ Performance on the Chinese Medical Licensing Examination: Quantitative Comparative Study},
  journal = {JMIR Human Factors},
  year = {2025},
  volume = {12},
  pages = {e77978--e77978},
  doi = {10.2196/77978}
}
Start for free →