Novel Evaluation Metric and Quantified Performance of ChatGPT-4 Patient Management Simulations for Early Clinical Education: Experimental Study
JMIR Formative Research · 27.02.2025 · 9 · e66478-e66478
DOI: 10.2196/66478 ↗Kurz gefasst
Fachartikel in der Zeitschrift JMIR Formative Research, online erschienen am 27.02.2025. Verfasst von Riley Scherr und 7 weiteren Autor:innen. Beteiligte Einrichtung: School of Medicine, University of California, Irvine School of Medicine, 1001 Health Sciences Road, Irvine, CA, 92617, United States, 1 9498246119. Hauptthemen laut MEDLINE: Patient Simulation, Clinical Competence, Simulation Training, Education, Medical, Undergraduate. Die Arbeit wurde bisher 14-mal zitiert (Crossref) und nennt 39 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Case studies have shown ChatGPT can run clinical simulations at the medical student level. However, no data have assessed ChatGPT’s reliability in meeting desired simulation criteria such as medical accuracy, simulation formatting, and robust feedback mechanisms.
Objective
This study aims to quantify ChatGPT’s ability to consistently follow formatting instructions and create simulations for preclinical medical student learners according to principles of medical simulation and multimedia educational technology.
Methods
Using ChatGPT-4 and a prevalidated starting prompt, the authors ran 360 separate simulations of an acute asthma exacerbation. A total of 180 simulations were given correct answers and 180 simulations were given incorrect answers. ChatGPT was evaluated for its ability to adhere to basic simulation parameters (stepwise progression, free response, interactivity), advanced simulation parameters (autonomous conclusion, delayed feedback, comprehensive feedback), and medical accuracy (vignette, treatment updates, feedback). Significance was determined with χ ² analyses using 95% CIs for odds ratios.
Results
In total, 100% (n=360) of simulations met basic simulation parameters and were medically accurate. For advanced parameters, 55% (200/360) of all simulations delayed feedback, while the Correct arm (157/180, 87%) delayed feedback was significantly more than the Incorrect arm (43/180, 24%; P
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- School of Medicine, University of California, Irvine School of Medicine, 1001 Health Sciences Road, Irvine, CA, 92617, United States, 1 9498246119.
- Department of Medicine, Stanford Medicine, Stanford, CA, United States.
- School of Medicine, Tufts University, Boston, MA, United States.
- Department of Emergency Medicine, University of California Irvine, Orange, CA, United States.
ORCID-Profile: Riley Scherr, Aidin Spina, Allen Dao, Saman Andalib, Faris F Halaseh, Sarah Blair, Warren Wiechmann, Ronald Rivera
Zitieren
APA
Scherr, R., Spina, A., Dao, A., Andalib, S., Halaseh, F. F., Blair, S., Wiechmann, W., & Rivera, R. (2025). Novel Evaluation Metric and Quantified Performance of ChatGPT-4 Patient Management Simulations for Early Clinical Education: Experimental Study. JMIR Formative Research, 9, e66478-e66478. https://doi.org/10.2196/66478
Vancouver
Scherr R, Spina A, Dao A, Andalib S, Halaseh FF, Blair S, et al. Novel Evaluation Metric and Quantified Performance of ChatGPT-4 Patient Management Simulations for Early Clinical Education: Experimental Study. JMIR Formative Research. 2025;9:e66478-e66478. doi:10.2196/66478
BibTeX
@article{Scherr2025,
author = {Scherr, Riley and Spina, Aidin and Dao, Allen and Andalib, Saman and Halaseh, Faris F and Blair, Sarah and Wiechmann, Warren and Rivera, Ronald},
title = {Novel Evaluation Metric and Quantified Performance of ChatGPT-4 Patient Management Simulations for Early Clinical Education: Experimental Study},
journal = {JMIR Formative Research},
year = {2025},
volume = {9},
pages = {e66478--e66478},
doi = {10.2196/66478}
}
EUVIMED