EUVIMEDEUVIMED entdecken
QuerschnittstudieOpen Access

Impact of Prompt Engineering on the Performance of ChatGPT Variants Across Different Question Types in Medical Student Examinations: Cross-Sectional Study

Ming-Yu Hsieh iD, Tzu-Ling Wang iD, Pen-Hua Su iD, Ming-Chih Chou iD

JMIR Medical Education · 01.10.2025 · 11 · e78320-e78320

DOI: 10.2196/78320 ↗

Kurz gefasst

Querschnittstudie in der Zeitschrift JMIR Medical Education, online erschienen am 01.10.2025. Verfasst von Ming-Yu Hsieh und 3 weiteren Autor:innen. Beteiligte Einrichtung: Division of Pediatric Surgery, Department of Surgery, Chung Shan Medical University Hospital, Taichung City, Taiwan. Hauptthemen laut MEDLINE: Students, Medical, Educational Measurement, Education, Medical, Undergraduate, Language. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 10 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models such as ChatGPT (OpenAI) have shown promise in medical education assessments, but the comparative effects of prompt engineering across optimized variants and relative performance against medical students remain unclear.

Objective

This study aims to systematically evaluate the impact of prompt engineering on five ChatGPT variants (GPT-3.5, GPT-4.0, GPT-4o, GPT-4o1-mini, and GPT-4o1) and benchmark their performance against fourth-year medical students in midterm and final examinations.

Methods

A 100-item examination dataset covering multiple choice questions, short answer questions, clinical case analysis, and image-based questions was administered to each model under no-prompt and prompt-engineering conditions over 5 independent runs. Student cohort scores (N=143) were collected for comparison. Responses were scored using standardized rubrics, converted to percentages, and analyzed in SPSS Statistics (v29.0) with paired t tests and Cohen d ( P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Students, MedicalEducational MeasurementEducation, Medical, UndergraduateLanguageHumansCross-Sectional StudiesFemaleMaleGenerative Artificial Intelligence

Stichwörter der Autor:innen

ChatGPTassessment performancelarge language modelsmedical educationprompt engineering

Autor:innen und Institutionen

  • Division of Pediatric Surgery, Department of Surgery, Chung Shan Medical University Hospital, Taichung City, Taiwan.
  • Institute of Medicine, School of Medicine, Chung Shan Medical University, No. 110, Sec. 2, Jiang Kuo South Road, South district, Taichung City, Taiwan, 886 4-24739595 ext 34601.
  • Department of Nursing, School of Medicine, Chung Shan Medical University, Taichung City, Taiwan.
  • Department of Pediatrics, Chung Shan Medical University Hospital, Taichung City, Taiwan.
  • Division of Thoracic Surgery, Department of Surgery, Chung Shan Medical University Hospital, Taichung City, Taiwan.

ORCID-Profile: Ming-Yu Hsieh, Tzu-Ling Wang, Pen-Hua Su, Ming-Chih Chou

Zitieren

APA

Hsieh, M. Y., Wang, T. L., Su, P. H., & Chou, M. C. (2025). Impact of Prompt Engineering on the Performance of ChatGPT Variants Across Different Question Types in Medical Student Examinations: Cross-Sectional Study. JMIR Medical Education, 11, e78320-e78320. https://doi.org/10.2196/78320

Vancouver

Hsieh MY, Wang TL, Su PH, Chou MC. Impact of Prompt Engineering on the Performance of ChatGPT Variants Across Different Question Types in Medical Student Examinations: Cross-Sectional Study. JMIR Medical Education. 2025;11:e78320-e78320. doi:10.2196/78320

BibTeX
@article{Hsieh2025,
  author = {Hsieh, Ming-Yu and Wang, Tzu-Ling and Su, Pen-Hua and Chou, Ming-Chih},
  title = {Impact of Prompt Engineering on the Performance of ChatGPT Variants Across Different Question Types in Medical Student Examinations: Cross-Sectional Study},
  journal = {JMIR Medical Education},
  year = {2025},
  volume = {11},
  pages = {e78320--e78320},
  doi = {10.2196/78320}
}
Kostenlos starten →