Comprehensive Evaluation of Large Language Models on Four Core Medical School Courses: A Cross-Sectional Comparative Study
Advances in Medical Education and Practice · 2026-09 · Volume 17 · 1-10
DOI: 10.2147/amep.s642751 ↗Kurz gefasst
Querschnittstudie in der Zeitschrift Advances in Medical Education and Practice, 2026-09. Verfasst von Kai Zhang und 2 weiteren Autor:innen. Beteiligte Einrichtung: College of Artificial Intelligence Medicine, Chongqing Medical University, Chongqing, People's Republic of China. Die Arbeit wurde bisher 1-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY-NC 3.0).
Abstract (Originalsprache)
Purpose
Large language models (LLMs) have demonstrated remarkable potential in medical education, yet their performance on discipline-specific medical school course examinations remains incompletely characterized. This study evaluated six contemporary LLMs on final examinations for four core medical school courses: Surgery, Musculoskeletal System Diseases, Digestive System Diseases, and Respiratory System Diseases.
Methods
A total of 400 multiple-choice questions (100 per course) were administered to each model. Responses were scored against official answer keys and compared to the performance of 312 medical students. Each question was tested three times per model to assess response consistency and reproducibility.
Results
All six LLMs achieved mean scores exceeding 93% across all four courses, substantially outperforming the student cohort mean of 71.8%, with all models scoring above the 99.5th percentile of the student score distribution. ChatGPT 5.5 achieved the highest overall accuracy (97.0%), followed by Claude Opus 4.7 (96.3%) and DeepSeek V4 (95.8%). Qwen 3.7 (95.3%) and GLM 5.2 (94.3%) demonstrated comparable performance, while Doubao 2.1 (93.8%) showed the lowest but still exceptional accuracy. A sensitivity analysis revealed that accuracy on the 12 visual-containing questions was substantially lower (50.0%) than on text-only questions (96.8%). All models demonstrated high response consistency (κ = 0.82-0.95) and reproducibility (>95%).
Conclusions
Contemporary LLMs can achieve near-perfect performance on medical school course examinations, substantially exceeding average student performance. While this capability suggests significant potential for LLMs as supplementary educational tools, it also raises important concerns regarding assessment integrity and the appropriate role of AI in medical training.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- College of Artificial Intelligence Medicine, Chongqing Medical University, Chongqing, People's Republic of China.
- School of Clinical Medicine, Chongqing Medical and Pharmaceutical College, Chongqing, People's Republic of China.
Zitieren
APA
Zhang, K., Yang, W., & Zheng, W. (2026). Comprehensive Evaluation of Large Language Models on Four Core Medical School Courses: A Cross-Sectional Comparative Study. Advances in Medical Education and Practice, Volume 17, 1-10. https://doi.org/10.2147/amep.s642751
Vancouver
Zhang K, Yang W, Zheng W. Comprehensive Evaluation of Large Language Models on Four Core Medical School Courses: A Cross-Sectional Comparative Study. Advances in Medical Education and Practice. 2026;Volume 17:1-10. doi:10.2147/amep.s642751
BibTeX
@article{Zhang2026,
author = {Zhang, Kai and Yang, Wenjing and Zheng, Wei},
title = {Comprehensive Evaluation of Large Language Models on Four Core Medical School Courses: A Cross-Sectional Comparative Study},
journal = {Advances in Medical Education and Practice},
year = {2026},
volume = {Volume 17},
pages = {1--10},
doi = {10.2147/amep.s642751}
}
EUVIMED