Impact of Prompt Engineering on the Performance of ChatGPT Variants Across Different Question Types in Medical Student Examinations: Cross-Sectional Study
JMIR Medical Education · 01.10.2025 · 11 · e78320-e78320
DOI: 10.2196/78320 ↗Kurz gefasst
Querschnittstudie in der Zeitschrift JMIR Medical Education, online erschienen am 01.10.2025. Verfasst von Ming-Yu Hsieh und 3 weiteren Autor:innen. Beteiligte Einrichtung: Division of Pediatric Surgery, Department of Surgery, Chung Shan Medical University Hospital, Taichung City, Taiwan. Hauptthemen laut MEDLINE: Students, Medical, Educational Measurement, Education, Medical, Undergraduate, Language. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 10 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Large language models such as ChatGPT (OpenAI) have shown promise in medical education assessments, but the comparative effects of prompt engineering across optimized variants and relative performance against medical students remain unclear.
Objective
This study aims to systematically evaluate the impact of prompt engineering on five ChatGPT variants (GPT-3.5, GPT-4.0, GPT-4o, GPT-4o1-mini, and GPT-4o1) and benchmark their performance against fourth-year medical students in midterm and final examinations.
Methods
A 100-item examination dataset covering multiple choice questions, short answer questions, clinical case analysis, and image-based questions was administered to each model under no-prompt and prompt-engineering conditions over 5 independent runs. Student cohort scores (N=143) were collected for comparison. Responses were scored using standardized rubrics, converted to percentages, and analyzed in SPSS Statistics (v29.0) with paired t tests and Cohen d ( P
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Division of Pediatric Surgery, Department of Surgery, Chung Shan Medical University Hospital, Taichung City, Taiwan.
- Institute of Medicine, School of Medicine, Chung Shan Medical University, No. 110, Sec. 2, Jiang Kuo South Road, South district, Taichung City, Taiwan, 886 4-24739595 ext 34601.
- Department of Nursing, School of Medicine, Chung Shan Medical University, Taichung City, Taiwan.
- Department of Pediatrics, Chung Shan Medical University Hospital, Taichung City, Taiwan.
- Division of Thoracic Surgery, Department of Surgery, Chung Shan Medical University Hospital, Taichung City, Taiwan.
ORCID-Profile: Ming-Yu Hsieh, Tzu-Ling Wang, Pen-Hua Su, Ming-Chih Chou
Zitieren
APA
Hsieh, M. Y., Wang, T. L., Su, P. H., & Chou, M. C. (2025). Impact of Prompt Engineering on the Performance of ChatGPT Variants Across Different Question Types in Medical Student Examinations: Cross-Sectional Study. JMIR Medical Education, 11, e78320-e78320. https://doi.org/10.2196/78320
Vancouver
Hsieh MY, Wang TL, Su PH, Chou MC. Impact of Prompt Engineering on the Performance of ChatGPT Variants Across Different Question Types in Medical Student Examinations: Cross-Sectional Study. JMIR Medical Education. 2025;11:e78320-e78320. doi:10.2196/78320
BibTeX
@article{Hsieh2025,
author = {Hsieh, Ming-Yu and Wang, Tzu-Ling and Su, Pen-Hua and Chou, Ming-Chih},
title = {Impact of Prompt Engineering on the Performance of ChatGPT Variants Across Different Question Types in Medical Student Examinations: Cross-Sectional Study},
journal = {JMIR Medical Education},
year = {2025},
volume = {11},
pages = {e78320--e78320},
doi = {10.2196/78320}
}
EUVIMED