EUVIMEDOpdag EUVIMED
Open access

Novel Evaluation Metric and Quantified Performance of ChatGPT-4 Patient Management Simulations for Early Clinical Education: Experimental Study

Riley Scherr iD, Aidin Spina iD, Allen Dao iD, Saman Andalib iD, Faris F Halaseh iD, Sarah Blair iD, Warren Wiechmann iD, Ronald Rivera iD

JMIR Formative Research · 2025-02-27 · 9 · e66478-e66478

DOI: 10.2196/66478 ↗

At a glance

Journal article in JMIR Formative Research, published online on 2025-02-27. Written by Riley Scherr and 7 co-authors. Institution: School of Medicine, University of California, Irvine School of Medicine, 1001 Health Sciences Road, Irvine, CA, 92617, United States, 1 9498246119. Major MEDLINE topics: Patient Simulation, Clinical Competence, Simulation Training, Education, Medical, Undergraduate. Cited 14 times so far (Crossref); 39 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

Case studies have shown ChatGPT can run clinical simulations at the medical student level. However, no data have assessed ChatGPT’s reliability in meeting desired simulation criteria such as medical accuracy, simulation formatting, and robust feedback mechanisms.

Objective

This study aims to quantify ChatGPT’s ability to consistently follow formatting instructions and create simulations for preclinical medical student learners according to principles of medical simulation and multimedia educational technology.

Methods

Using ChatGPT-4 and a prevalidated starting prompt, the authors ran 360 separate simulations of an acute asthma exacerbation. A total of 180 simulations were given correct answers and 180 simulations were given incorrect answers. ChatGPT was evaluated for its ability to adhere to basic simulation parameters (stepwise progression, free response, interactivity), advanced simulation parameters (autonomous conclusion, delayed feedback, comprehensive feedback), and medical accuracy (vignette, treatment updates, feedback). Significance was determined with χ ² analyses using 95% CIs for odds ratios.

Results

In total, 100% (n=360) of simulations met basic simulation parameters and were medically accurate. For advanced parameters, 55% (200/360) of all simulations delayed feedback, while the Correct arm (157/180, 87%) delayed feedback was significantly more than the Incorrect arm (43/180, 24%; P

Subjects

MeSH terms (MEDLINE)

Patient SimulationClinical CompetenceSimulation TrainingEducation, Medical, UndergraduateHumansReproducibility of ResultsStudents, MedicalEducational MeasurementFormative FeedbackGenerative Artificial Intelligence

Author keywords

AI in medical educationChatGPTChatGPT-4clinical educationfeedbackmedical educationmedical school simulationsmedical simulationmedical studentmultimediapatient managementpilot studypreclinical curriculumsimulation

Authors and institutions

  • School of Medicine, University of California, Irvine School of Medicine, 1001 Health Sciences Road, Irvine, CA, 92617, United States, 1 9498246119.
  • Department of Medicine, Stanford Medicine, Stanford, CA, United States.
  • School of Medicine, Tufts University, Boston, MA, United States.
  • Department of Emergency Medicine, University of California Irvine, Orange, CA, United States.

ORCID profiles: Riley Scherr, Aidin Spina, Allen Dao, Saman Andalib, Faris F Halaseh, Sarah Blair, Warren Wiechmann, Ronald Rivera

Cite

APA

Scherr, R., Spina, A., Dao, A., Andalib, S., Halaseh, F. F., Blair, S., Wiechmann, W., & Rivera, R. (2025). Novel Evaluation Metric and Quantified Performance of ChatGPT-4 Patient Management Simulations for Early Clinical Education: Experimental Study. JMIR Formative Research, 9, e66478-e66478. https://doi.org/10.2196/66478

Vancouver

Scherr R, Spina A, Dao A, Andalib S, Halaseh FF, Blair S, et al. Novel Evaluation Metric and Quantified Performance of ChatGPT-4 Patient Management Simulations for Early Clinical Education: Experimental Study. JMIR Formative Research. 2025;9:e66478-e66478. doi:10.2196/66478

BibTeX
@article{Scherr2025,
  author = {Scherr, Riley and Spina, Aidin and Dao, Allen and Andalib, Saman and Halaseh, Faris F and Blair, Sarah and Wiechmann, Warren and Rivera, Ronald},
  title = {Novel Evaluation Metric and Quantified Performance of ChatGPT-4 Patient Management Simulations for Early Clinical Education: Experimental Study},
  journal = {JMIR Formative Research},
  year = {2025},
  volume = {9},
  pages = {e66478--e66478},
  doi = {10.2196/66478}
}
Start for free →