Novel Evaluation Metric and Quantified Performance of ChatGPT-4 Patient Management Simulations for Early Clinical Education: Experimental Study
JMIR Formative Research · 2025-02-27 · 9 · e66478-e66478
DOI: 10.2196/66478 ↗At a glance
Journal article in JMIR Formative Research, published online on 2025-02-27. Written by Riley Scherr and 7 co-authors. Institution: School of Medicine, University of California, Irvine School of Medicine, 1001 Health Sciences Road, Irvine, CA, 92617, United States, 1 9498246119. Major MEDLINE topics: Patient Simulation, Clinical Competence, Simulation Training, Education, Medical, Undergraduate. Cited 14 times so far (Crossref); 39 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
Case studies have shown ChatGPT can run clinical simulations at the medical student level. However, no data have assessed ChatGPT’s reliability in meeting desired simulation criteria such as medical accuracy, simulation formatting, and robust feedback mechanisms.
Objective
This study aims to quantify ChatGPT’s ability to consistently follow formatting instructions and create simulations for preclinical medical student learners according to principles of medical simulation and multimedia educational technology.
Methods
Using ChatGPT-4 and a prevalidated starting prompt, the authors ran 360 separate simulations of an acute asthma exacerbation. A total of 180 simulations were given correct answers and 180 simulations were given incorrect answers. ChatGPT was evaluated for its ability to adhere to basic simulation parameters (stepwise progression, free response, interactivity), advanced simulation parameters (autonomous conclusion, delayed feedback, comprehensive feedback), and medical accuracy (vignette, treatment updates, feedback). Significance was determined with χ ² analyses using 95% CIs for odds ratios.
Results
In total, 100% (n=360) of simulations met basic simulation parameters and were medically accurate. For advanced parameters, 55% (200/360) of all simulations delayed feedback, while the Correct arm (157/180, 87%) delayed feedback was significantly more than the Incorrect arm (43/180, 24%; P
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- School of Medicine, University of California, Irvine School of Medicine, 1001 Health Sciences Road, Irvine, CA, 92617, United States, 1 9498246119.
- Department of Medicine, Stanford Medicine, Stanford, CA, United States.
- School of Medicine, Tufts University, Boston, MA, United States.
- Department of Emergency Medicine, University of California Irvine, Orange, CA, United States.
ORCID profiles: Riley Scherr, Aidin Spina, Allen Dao, Saman Andalib, Faris F Halaseh, Sarah Blair, Warren Wiechmann, Ronald Rivera
Cite
APA
Scherr, R., Spina, A., Dao, A., Andalib, S., Halaseh, F. F., Blair, S., Wiechmann, W., & Rivera, R. (2025). Novel Evaluation Metric and Quantified Performance of ChatGPT-4 Patient Management Simulations for Early Clinical Education: Experimental Study. JMIR Formative Research, 9, e66478-e66478. https://doi.org/10.2196/66478
Vancouver
Scherr R, Spina A, Dao A, Andalib S, Halaseh FF, Blair S, et al. Novel Evaluation Metric and Quantified Performance of ChatGPT-4 Patient Management Simulations for Early Clinical Education: Experimental Study. JMIR Formative Research. 2025;9:e66478-e66478. doi:10.2196/66478
BibTeX
@article{Scherr2025,
author = {Scherr, Riley and Spina, Aidin and Dao, Allen and Andalib, Saman and Halaseh, Faris F and Blair, Sarah and Wiechmann, Warren and Rivera, Ronald},
title = {Novel Evaluation Metric and Quantified Performance of ChatGPT-4 Patient Management Simulations for Early Clinical Education: Experimental Study},
journal = {JMIR Formative Research},
year = {2025},
volume = {9},
pages = {e66478--e66478},
doi = {10.2196/66478}
}
EUVIMED