Impact of Prompt Engineering on the Performance of ChatGPT Variants Across Different Question Types in Medical Student Examinations: Cross-Sectional Study
JMIR Medical Education · 2025-10-01 · 11 · e78320-e78320
DOI: 10.2196/78320 ↗At a glance
Querschnittstudie in JMIR Medical Education, published online on 2025-10-01. Written by Ming-Yu Hsieh and 3 co-authors. Institution: Division of Pediatric Surgery, Department of Surgery, Chung Shan Medical University Hospital, Taichung City, Taiwan. Major MEDLINE topics: Students, Medical, Educational Measurement, Education, Medical, Undergraduate, Language. Cited 13 times so far (Crossref); 10 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
Large language models such as ChatGPT (OpenAI) have shown promise in medical education assessments, but the comparative effects of prompt engineering across optimized variants and relative performance against medical students remain unclear.
Objective
This study aims to systematically evaluate the impact of prompt engineering on five ChatGPT variants (GPT-3.5, GPT-4.0, GPT-4o, GPT-4o1-mini, and GPT-4o1) and benchmark their performance against fourth-year medical students in midterm and final examinations.
Methods
A 100-item examination dataset covering multiple choice questions, short answer questions, clinical case analysis, and image-based questions was administered to each model under no-prompt and prompt-engineering conditions over 5 independent runs. Student cohort scores (N=143) were collected for comparison. Responses were scored using standardized rubrics, converted to percentages, and analyzed in SPSS Statistics (v29.0) with paired t tests and Cohen d ( P
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Division of Pediatric Surgery, Department of Surgery, Chung Shan Medical University Hospital, Taichung City, Taiwan.
- Institute of Medicine, School of Medicine, Chung Shan Medical University, No. 110, Sec. 2, Jiang Kuo South Road, South district, Taichung City, Taiwan, 886 4-24739595 ext 34601.
- Department of Nursing, School of Medicine, Chung Shan Medical University, Taichung City, Taiwan.
- Department of Pediatrics, Chung Shan Medical University Hospital, Taichung City, Taiwan.
- Division of Thoracic Surgery, Department of Surgery, Chung Shan Medical University Hospital, Taichung City, Taiwan.
ORCID profiles: Ming-Yu Hsieh, Tzu-Ling Wang, Pen-Hua Su, Ming-Chih Chou
Cite
APA
Hsieh, M. Y., Wang, T. L., Su, P. H., & Chou, M. C. (2025). Impact of Prompt Engineering on the Performance of ChatGPT Variants Across Different Question Types in Medical Student Examinations: Cross-Sectional Study. JMIR Medical Education, 11, e78320-e78320. https://doi.org/10.2196/78320
Vancouver
Hsieh MY, Wang TL, Su PH, Chou MC. Impact of Prompt Engineering on the Performance of ChatGPT Variants Across Different Question Types in Medical Student Examinations: Cross-Sectional Study. JMIR Medical Education. 2025;11:e78320-e78320. doi:10.2196/78320
BibTeX
@article{Hsieh2025,
author = {Hsieh, Ming-Yu and Wang, Tzu-Ling and Su, Pen-Hua and Chou, Ming-Chih},
title = {Impact of Prompt Engineering on the Performance of ChatGPT Variants Across Different Question Types in Medical Student Examinations: Cross-Sectional Study},
journal = {JMIR Medical Education},
year = {2025},
volume = {11},
pages = {e78320--e78320},
doi = {10.2196/78320}
}
EUVIMED