EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access · CC BY 4.0

Performance of DeepSeek and ChatGPT on the Chinese Health Professional and Technical Examination: A comparative study

Xu Li iD, Xu Hu, Huiting Xu, Zhiang Sun, Pin Yu, Hailing Ju

PLOS One · 2026

DOI: 10.1371/journal.pone.0338328 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift PLOS One, 2026. Verfasst von Xu Li und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Central ICU, The First Affiliated Hospital of Soochow University, Suzhou, China. Hauptthemen laut MEDLINE: Educational Measurement, Health Personnel. Die Arbeit nennt 34 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: "Zhou's" nursing research project of The First Affiliated Hospital of Soochow University, University-Industry Collaborative Education Program.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models (LLMs) are increasingly applied in medical education, yet their reliability in specialized, high-stakes assessments such as the Chinese Health Professional and Technical Examination remains unclear. DeepSeek-R1, a recently released reasoning-enhanced LLM, has shown promising performance, but empirical evidence within nursing examination contexts is limited.

Objective

To compare the performance of DeepSeek-R1 and the GPT-4o API on the Chinese Health Professional and Technical Examination (Intermediate Nursing), focusing on accuracy, response consistency, and consistent accuracy.

Methods

Four hundred official practice examination multiple-choice questions were categorized into four competency units and two question types (A/B). Both models were evaluated using overall accuracy, consistency (agreement across repeated responses), and consistent accuracy (proportion of responses that were both consistent and correct). Stratified analyses were performed across units, question types, and disciplines. Chi-square tests were used for statistical comparison, and Holm–Bonferroni correction was applied for multiple comparisons.

Results

DeepSeek-R1 demonstrated significantly higher overall accuracy than the GPT-4o API (88.5% vs. 67.9%, P < 0.001). GPT-4o API showed higher response consistency (96.5% vs. 88.5%) but lower consistent accuracy (66.7% vs. 84.0%). After multiple-comparison correction, significant differences in consistent accuracy remained in basic knowledge, professional knowledge, professional practice ability and Type A questions, as well as in surgical and gynecological nursing disciplines, while other domains showed no statistically significant differences.

Conclusion

DeepSeek-R1 outperformed the GPT-4o API across multiple dimensions of nursing competency assessment, particularly in overall accuracy and consistent accuracy. GPT-4o API exhibited high response stability but a tendency toward systematic errors, underscoring the need for careful interpretation of model outputs. Further research is needed to evaluate LLM performance using open-ended clinical reasoning tasks and real-world assessment data to support safe and effective educational integration.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Educational MeasurementHealth PersonnelLarge Language ModelsHumansChinaGenerative Artificial IntelligenceReproducibility of Results

Autor:innen und Institutionen

  • Department of Central ICU, The First Affiliated Hospital of Soochow University, Suzhou, China.
  • School of Medicine, Tongji University, Shanghai, China.
  • Department of Operating room, Qingpu Branch, Zhongshan Hospital Affiliated to Fudan University, Shanghai, China.
  • Department of Nursing, Tenth People's Hospital Affiliated to Tongji University, Shanghai, China.

ORCID-Profile: Xu Li

Förderung

  • "Zhou's" nursing research project of The First Affiliated Hospital of Soochow University
  • University-Industry Collaborative Education Program

Zitieren

APA

Li, X., Hu, X., Xu, H., Sun, Z., Yu, P., & Ju, H. (2026). Performance of DeepSeek and ChatGPT on the Chinese Health Professional and Technical Examination: A comparative study. PLOS One. https://doi.org/10.1371/journal.pone.0338328

Vancouver

Li X, Hu X, Xu H, Sun Z, Yu P, Ju H. Performance of DeepSeek and ChatGPT on the Chinese Health Professional and Technical Examination: A comparative study. PLOS One. 2026. doi:10.1371/journal.pone.0338328

BibTeX
@article{Li2026,
  author = {Li, Xu and Hu, Xu and Xu, Huiting and Sun, Zhiang and Yu, Pin and Ju, Hailing},
  title = {Performance of DeepSeek and ChatGPT on the Chinese Health Professional and Technical Examination: A comparative study},
  journal = {PLOS One},
  year = {2026},
  doi = {10.1371/journal.pone.0338328}
}
Kostenlos starten →