EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access

Multiple Large Language Models’ Performance on the Chinese Medical Licensing Examination: Quantitative Comparative Study

Yanyu Diao iD, Mengyuan Wu iD, Jingwen Xu iD, Yifeng Pan iD

JMIR Human Factors · 16.12.2025 · 12 · e77978-e77978

DOI: 10.2196/77978 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift JMIR Human Factors, online erschienen am 16.12.2025. Verfasst von Yanyu Diao und 3 weiteren Autor:innen. Beteiligte Einrichtung: The School of Big Data and Artificial Intelligence, Anhui Xinhua University, 555 Wangjiang West Road, Hefei, 230088, China, 86 15905667742. Hauptthemen laut MEDLINE: Language, Educational Measurement, Licensure, Medical, Education, Medical. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 19 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

ChatGPT excels in natural language tasks, but its performance in the Chinese National Medical Licensing Examination (NMLE) and Chinese medical education remains underexplored. Meanwhile, Chinese corpus–based large language models (LLMs) such as ERNIE Bot, Tongyi Qianwen, Doubao, and DeepSeek have emerged, yet their effectiveness in the NMLE awaits systematic evaluation.

Objective

This study aimed to quantitatively compare the performance of 6 LLMs (GPT-3.5, GPT-4, ERNIE Bot, Tongyi Qianwen, Doubao, and DeepSeek) in answering NMLE questions from 2018 to 2024 and analyze their feasibility as supplementary tools in Chinese medical education.

Methods

We selected questions from the 4 content units of the NMLE’s General Written test (2018‐2024), preprocessed image- and table-based content into standardized text, and input the questions into each model. We evaluated the accuracy, comprehensiveness, and logical coherence of the responses, with quantitative comparison centered on scores and accuracy rates against the official answer keys (passing score: 360/600).

Results

GPT-4 outperformed GPT-3.5 across all units, achieving average accuracies of 66.57% (SD 3.21%; unit 1), 69.05% (SD 2.87%; unit 2), 71.71% (SD 2.53%; unit 3), and 80.67% (SD 2.19%; unit 4), with consistent scores above the passing threshold. Among the Chinese models, DeepSeek demonstrated the highest overall performance, with an average score of 454.8 (SD 17.3) and average accuracies of 73.2% (unit 1, SD 2.89%) and 71.5% (unit 3, SD 2.64%), as well as average accuracies of 70.3% (unit 2, SD 3.02%) and 78.2% (unit 4, SD 2.47%). ERNIE Bot (mean score 442.3, SD 19.6; unit 1 accuracy =70.8%, SD 3.01%; unit 2 accuracy =68.7%, SD 3.15%; unit 3 accuracy =69.1%, SD 2.93%; unit 4 accuracy =68.3%, SD 2.76%), Tongyi Qianwen (mean score 426.5, SD 21.4; unit 1 accuracy =67.4%, SD 3.22%; unit 2 accuracy =65.9%, SD 3.31%; unit 3 accuracy =66.2%, SD 3.08%; unit 4 accuracy =67.2%, SD 2.89%), and Doubao (mean score 413.7, SD 23.1; unit 1 accuracy =65.2%, SD 3.45%; unit 2 accuracy =63.8%, SD 3.52%; unit 3 accuracy =64.1%, SD 3.27%; unit 4 accuracy =62.8%, SD 3.11%) all exceeded the passing score. DeepSeek’s overall average accuracy (75.8%, SD 2.73%) was significantly higher than those of the other Chinese models (χ²₁=11.4, P=.001 vs ERNIE Bot; χ²₁=28.7, P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

LanguageEducational MeasurementLicensure, MedicalEducation, MedicalHumansChinaLarge Language Models

Stichwörter der Autor:innen

AIChatGPTChinese National Medical Licensing ExaminationERNIE BotTongyi Qianwenartificial intelligencemedical student

Autor:innen und Institutionen

  • The School of Big Data and Artificial Intelligence, Anhui Xinhua University, 555 Wangjiang West Road, Hefei, 230088, China, 86 15905667742.

ORCID-Profile: Yanyu Diao, Mengyuan Wu, Jingwen Xu, Yifeng Pan

Zitieren

APA

Diao, Y., Wu, M., Xu, J., & Pan, Y. (2025). Multiple Large Language Models’ Performance on the Chinese Medical Licensing Examination: Quantitative Comparative Study. JMIR Human Factors, 12, e77978-e77978. https://doi.org/10.2196/77978

Vancouver

Diao Y, Wu M, Xu J, Pan Y. Multiple Large Language Models’ Performance on the Chinese Medical Licensing Examination: Quantitative Comparative Study. JMIR Human Factors. 2025;12:e77978-e77978. doi:10.2196/77978

BibTeX
@article{Diao2025,
  author = {Diao, Yanyu and Wu, Mengyuan and Xu, Jingwen and Pan, Yifeng},
  title = {Multiple Large Language Models’ Performance on the Chinese Medical Licensing Examination: Quantitative Comparative Study},
  journal = {JMIR Human Factors},
  year = {2025},
  volume = {12},
  pages = {e77978--e77978},
  doi = {10.2196/77978}
}
Kostenlos starten →