Comparative analysis between the performance of ChatGPT and medical students outside their examination phase
GMS Medizinische Informatik, Biometrie und Epidemiologie · 25.03.2026 · 22 (Doc05) · Doc05
DOI: 10.3205/mibe000303 ↗Kurz gefasst
Fachartikel in der Zeitschrift GMS Medizinische Informatik, Biometrie und Epidemiologie, erschienen am 25.03.2026. Verfasst von Leufkens und 7 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Introduction
State-run medical licensing examinations in Germany are highly demanding, requiring extensive preparation. While medical expertise remains central to licensure, patients increasingly seek advice from artificial intelligence (AI) models like ChatGPT. This shift raises the question of whether AI can accurately convey essential medical knowledge. Previous international studies (e.g., China, US, Poland, UK) have compared AI performance with that of certified professionals, often through indirect comparisons using historical exam averages. However, no studies have directly compared AI to medical students assessed outside their exam preparation phase, which could provide insight into knowledge retention. This study aims to directly compare the performance of large language models (LLMs) with that of medical students beyond their examination phase.
Methods
An anonymized survey was conducted at a German medical school among students in the clinical stage of their studies (typically 170 to 180 students). Participants answered 10 single-choice questions randomly selected from a pre-filtered pool derived from past German preclinical medical exam (M1) items. Questions were selected based on clinical relevance, moderate difficulty, and exclusion of chemical/mathematical content. The same questions were answered by ChatGPT-3.5, ChatGPT-4, and ChatGPT-4 mini. Performance was compared in terms of correct responses. Additionally, the corrected discrimination coefficient was calculated for each item, measuring how well each question differentiated between higher and lower performers.
Results
Of the 143 participants (median age 22), 129 were in the 5th semester, and the rest were in later semesters. About 40% identified as male and 55% as female. Students answered a median of 7 out of 10 questions correctly (range: 1–10). All AI models answered 9 out of 10 questions correctly. The only question missed by AI was answered correctly by 35% of students (50/143) and had the second-highest discrimination coefficient (0.28), indicating it effectively differentiated student performance.
Discussion
LLMs outperformed medical students who were beyond their exam preparation phase. However, limitations include the modest sample size and preselection of questions based on specific criteria (clinical relevance, moderate difficulty, exclusion of chemical/mathematical content), which introduces selection bias and limits the representativeness of our findings for the full examination. Importantly, the AI’s one incorrect answer had the second highest, although still marginal discrimination coefficient, highlighting a possible gap in AI understanding for nuanced or complex content. These findings suggest that AI models are not without limitations and should be supplemented by human oversight, particularly in high-stakes or ambiguous clinical contexts.
Conclusion
AI models demonstrate strong performance in answering single choice questions, surpassing students outside active exam preparation. However, their occasional errors, especially on discriminative questions, underline the need for caution. Further research is necessary to evaluate AI utility in real-world medical education and clinical decision-making, ensuring ethical and responsible integration.
Schlagwörter
Zitieren
APA
Leufkens, Daniel, Pons-Kühnemann, Jörn, Schneider, Henning, Windhorst, & C., A. (2026). Comparative analysis between the performance of ChatGPT and medical students outside their examination phase. GMS Medizinische Informatik, Biometrie und Epidemiologie, 22(Doc05), Doc05. https://doi.org/10.3205/mibe000303
Vancouver
Leufkens, Daniel, Pons-Kühnemann, Jörn, Schneider, Henning, et al. Comparative analysis between the performance of ChatGPT and medical students outside their examination phase. GMS Medizinische Informatik, Biometrie und Epidemiologie. 2026;22(Doc05):Doc05. doi:10.3205/mibe000303
BibTeX
@article{Leufkens2026,
author = {Leufkens and Daniel and Pons-Kühnemann and Jörn and Schneider and Henning and Windhorst and C., Anita},
title = {Comparative analysis between the performance of ChatGPT and medical students outside their examination phase},
journal = {GMS Medizinische Informatik, Biometrie und Epidemiologie},
year = {2026},
volume = {22},
number = {Doc05},
pages = {Doc05},
doi = {10.3205/mibe000303}
}
EUVIMED