EUVIMEDObjevte EUVIMED
QuerschnittstudieOpen access

Impact of Prompt Engineering on the Performance of ChatGPT Variants Across Different Question Types in Medical Student Examinations: Cross-Sectional Study

Ming-Yu Hsieh iD, Tzu-Ling Wang iD, Pen-Hua Su iD, Ming-Chih Chou iD

JMIR Medical Education · 2025-10-01 · 11 · e78320-e78320

DOI: 10.2196/78320 ↗

At a glance

Querschnittstudie in JMIR Medical Education, published online on 2025-10-01. Written by Ming-Yu Hsieh and 3 co-authors. Institution: Division of Pediatric Surgery, Department of Surgery, Chung Shan Medical University Hospital, Taichung City, Taiwan. Major MEDLINE topics: Students, Medical, Educational Measurement, Education, Medical, Undergraduate, Language. Cited 13 times so far (Crossref); 10 references. The full text is freely available in PubMed Central.

AbstraktSubjectsBibliografické údajeCiteRelated studies

Abstract (original language)

Background

Large language models such as ChatGPT (OpenAI) have shown promise in medical education assessments, but the comparative effects of prompt engineering across optimized variants and relative performance against medical students remain unclear.

Objective

This study aims to systematically evaluate the impact of prompt engineering on five ChatGPT variants (GPT-3.5, GPT-4.0, GPT-4o, GPT-4o1-mini, and GPT-4o1) and benchmark their performance against fourth-year medical students in midterm and final examinations.

Methods

A 100-item examination dataset covering multiple choice questions, short answer questions, clinical case analysis, and image-based questions was administered to each model under no-prompt and prompt-engineering conditions over 5 independent runs. Student cohort scores (N=143) were collected for comparison. Responses were scored using standardized rubrics, converted to percentages, and analyzed in SPSS Statistics (v29.0) with paired t tests and Cohen d ( P

Subjects

MeSH terms (MEDLINE)

Students, MedicalEducational MeasurementEducation, Medical, UndergraduateLanguageHumansCross-Sectional StudiesFemaleMaleGenerative Artificial Intelligence

Author keywords

ChatGPTassessment performancelarge language modelsmedical educationprompt engineering

Authors and institutions

  • Division of Pediatric Surgery, Department of Surgery, Chung Shan Medical University Hospital, Taichung City, Taiwan.
  • Institute of Medicine, School of Medicine, Chung Shan Medical University, No. 110, Sec. 2, Jiang Kuo South Road, South district, Taichung City, Taiwan, 886 4-24739595 ext 34601.
  • Department of Nursing, School of Medicine, Chung Shan Medical University, Taichung City, Taiwan.
  • Department of Pediatrics, Chung Shan Medical University Hospital, Taichung City, Taiwan.
  • Division of Thoracic Surgery, Department of Surgery, Chung Shan Medical University Hospital, Taichung City, Taiwan.

ORCID profiles: Ming-Yu Hsieh, Tzu-Ling Wang, Pen-Hua Su, Ming-Chih Chou

Cite

APA

Hsieh, M. Y., Wang, T. L., Su, P. H., & Chou, M. C. (2025). Impact of Prompt Engineering on the Performance of ChatGPT Variants Across Different Question Types in Medical Student Examinations: Cross-Sectional Study. JMIR Medical Education, 11, e78320-e78320. https://doi.org/10.2196/78320

Vancouver

Hsieh MY, Wang TL, Su PH, Chou MC. Impact of Prompt Engineering on the Performance of ChatGPT Variants Across Different Question Types in Medical Student Examinations: Cross-Sectional Study. JMIR Medical Education. 2025;11:e78320-e78320. doi:10.2196/78320

BibTeX
@article{Hsieh2025,
  author = {Hsieh, Ming-Yu and Wang, Tzu-Ling and Su, Pen-Hua and Chou, Ming-Chih},
  title = {Impact of Prompt Engineering on the Performance of ChatGPT Variants Across Different Question Types in Medical Student Examinations: Cross-Sectional Study},
  journal = {JMIR Medical Education},
  year = {2025},
  volume = {11},
  pages = {e78320--e78320},
  doi = {10.2196/78320}
}
Start for free →