EUVIMEDEUVIMED entdecken
Open Access

Novel Evaluation Metric and Quantified Performance of ChatGPT-4 Patient Management Simulations for Early Clinical Education: Experimental Study

Riley Scherr iD, Aidin Spina iD, Allen Dao iD, Saman Andalib iD, Faris F Halaseh iD, Sarah Blair iD, Warren Wiechmann iD, Ronald Rivera iD

JMIR Formative Research · 27.02.2025 · 9 · e66478-e66478

DOI: 10.2196/66478 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Formative Research, online erschienen am 27.02.2025. Verfasst von Riley Scherr und 7 weiteren Autor:innen. Beteiligte Einrichtung: School of Medicine, University of California, Irvine School of Medicine, 1001 Health Sciences Road, Irvine, CA, 92617, United States, 1 9498246119. Hauptthemen laut MEDLINE: Patient Simulation, Clinical Competence, Simulation Training, Education, Medical, Undergraduate. Die Arbeit wurde bisher 14-mal zitiert (Crossref) und nennt 39 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Case studies have shown ChatGPT can run clinical simulations at the medical student level. However, no data have assessed ChatGPT’s reliability in meeting desired simulation criteria such as medical accuracy, simulation formatting, and robust feedback mechanisms.

Objective

This study aims to quantify ChatGPT’s ability to consistently follow formatting instructions and create simulations for preclinical medical student learners according to principles of medical simulation and multimedia educational technology.

Methods

Using ChatGPT-4 and a prevalidated starting prompt, the authors ran 360 separate simulations of an acute asthma exacerbation. A total of 180 simulations were given correct answers and 180 simulations were given incorrect answers. ChatGPT was evaluated for its ability to adhere to basic simulation parameters (stepwise progression, free response, interactivity), advanced simulation parameters (autonomous conclusion, delayed feedback, comprehensive feedback), and medical accuracy (vignette, treatment updates, feedback). Significance was determined with χ ² analyses using 95% CIs for odds ratios.

Results

In total, 100% (n=360) of simulations met basic simulation parameters and were medically accurate. For advanced parameters, 55% (200/360) of all simulations delayed feedback, while the Correct arm (157/180, 87%) delayed feedback was significantly more than the Incorrect arm (43/180, 24%; P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Patient SimulationClinical CompetenceSimulation TrainingEducation, Medical, UndergraduateHumansReproducibility of ResultsStudents, MedicalEducational MeasurementFormative FeedbackGenerative Artificial Intelligence

Stichwörter der Autor:innen

AI in medical educationChatGPTChatGPT-4clinical educationfeedbackmedical educationmedical school simulationsmedical simulationmedical studentmultimediapatient managementpilot studypreclinical curriculumsimulation

Autor:innen und Institutionen

  • School of Medicine, University of California, Irvine School of Medicine, 1001 Health Sciences Road, Irvine, CA, 92617, United States, 1 9498246119.
  • Department of Medicine, Stanford Medicine, Stanford, CA, United States.
  • School of Medicine, Tufts University, Boston, MA, United States.
  • Department of Emergency Medicine, University of California Irvine, Orange, CA, United States.

ORCID-Profile: Riley Scherr, Aidin Spina, Allen Dao, Saman Andalib, Faris F Halaseh, Sarah Blair, Warren Wiechmann, Ronald Rivera

Zitieren

APA

Scherr, R., Spina, A., Dao, A., Andalib, S., Halaseh, F. F., Blair, S., Wiechmann, W., & Rivera, R. (2025). Novel Evaluation Metric and Quantified Performance of ChatGPT-4 Patient Management Simulations for Early Clinical Education: Experimental Study. JMIR Formative Research, 9, e66478-e66478. https://doi.org/10.2196/66478

Vancouver

Scherr R, Spina A, Dao A, Andalib S, Halaseh FF, Blair S, et al. Novel Evaluation Metric and Quantified Performance of ChatGPT-4 Patient Management Simulations for Early Clinical Education: Experimental Study. JMIR Formative Research. 2025;9:e66478-e66478. doi:10.2196/66478

BibTeX
@article{Scherr2025,
  author = {Scherr, Riley and Spina, Aidin and Dao, Allen and Andalib, Saman and Halaseh, Faris F and Blair, Sarah and Wiechmann, Warren and Rivera, Ronald},
  title = {Novel Evaluation Metric and Quantified Performance of ChatGPT-4 Patient Management Simulations for Early Clinical Education: Experimental Study},
  journal = {JMIR Formative Research},
  year = {2025},
  volume = {9},
  pages = {e66478--e66478},
  doi = {10.2196/66478}
}
Kostenlos starten →