EUVIMEDSkopri EUVIMED
StudienprotokollOpen access

Accuracy of a Commercial Large Language Model (ChatGPT) to Perform Disaster Triage of Simulated Patients Using the Simple Triage and Rapid Treatment (START) Protocol: Gage Repeatability and Reproducibility Study

Jeffrey Micheal Franc iD, Attila Julius Hertelendy iD, Lenard Cheng iD, Ryan Hata iD, Manuela Verde iD

Journal of Medical Internet Research · 2024-09-30 · 26 · e55648

DOI: 10.2196/55648 ↗

At a glance

Studienprotokoll in Journal of Medical Internet Research, published online on 2024-09-30. Written by Jeffrey Micheal Franc and 4 co-authors. Institution: Department of Emergency Medicine, University of Alberta, Edmonton, AB, Canada. Major MEDLINE topics: Triage. Cited 36 times so far (Crossref); 26 references. The full text is freely available in PubMed Central.

AstrattSubjectsDettalji biblijografiċiCiteRelated studies

Abstract (original language)

Background

The release of ChatGPT (OpenAI) in November 2022 drastically reduced the barrier to using artificial intelligence by allowing a simple web-based text interface to a large language model (LLM). One use case where ChatGPT could be useful is in triaging patients at the site of a disaster using the Simple Triage and Rapid Treatment (START) protocol. However, LLMs experience several common errors including hallucinations (also called confabulations) and prompt dependency.

Objective

This study addresses the research problem: “Can ChatGPT adequately triage simulated disaster patients using the START protocol?” by measuring three outcomes: repeatability, reproducibility, and accuracy.

Methods

Nine prompts were developed by 5 disaster medicine physicians. A Python script queried ChatGPT Version 4 for each prompt combined with 391 validated simulated patient vignettes. Ten repetitions of each combination were performed for a total of 35,190 simulated triages. A reference standard START triage code for each simulated case was assigned by 2 disaster medicine specialists (JMF and MV), with a third specialist (LC) added if the first two did not agree. Results were evaluated using a gage repeatability and reproducibility study (gage R and R). Repeatability was defined as variation due to repeated use of the same prompt. Reproducibility was defined as variation due to the use of different prompts on the same patient vignette. Accuracy was defined as agreement with the reference standard.

Results

Although 35,102 (99.7%) queries returned a valid START score, there was considerable variability. Repeatability (use of the same prompt repeatedly) was 14% of the overall variation. Reproducibility (use of different prompts) was 4.1% of the overall variation. The accuracy of ChatGPT for START was 63.9% with a 32.9% overtriage rate and a 3.1% undertriage rate. Accuracy varied by prompt with a maximum of 71.8% and a minimum of 46.7%.

Conclusions

This study indicates that ChatGPT version 4 is insufficient to triage simulated disaster patients via the START protocol. It demonstrated suboptimal repeatability and reproducibility. The overall accuracy of triage was only 63.9%. Health care professionals are advised to exercise caution while using commercial LLMs for vital medical determinations, given that these tools may commonly produce inaccurate data, colloquially referred to as hallucinations or confabulations. Artificial intelligence–guided tools should undergo rigorous statistical evaluation—using methods such as gage R and R—before implementation into clinical settings.

Subjects

MeSH terms (MEDLINE)

TriageHumansReproducibility of ResultsPatient SimulationDisaster MedicineDisasters

Author keywords

ChatGPTGPTLLMLLMsNLPaccuracyaccurateartificial intelligencedisasterdisaster medicinedisastersemergenciesemergencylanguage modellanguage models

Authors and institutions

  • Department of Emergency Medicine, University of Alberta, Edmonton, AB, Canada.
  • CRIMEDIM-Center for Research and Training in Disaster Medicine, Humanitarian Aid and Global Health, Universita' del Piemonte Orientale, Novara, Italy.
  • Department of Information Systems and Business Analytics, College of Business, Florida International University, Miami, FL, United States.
  • Department of Emergency Medicine, Beth Isreal Deaconess Medical Center, Harvard Medical School Teaching Hospital, Boston, MA, United States.
  • Emergency Medicine Department, National University Hospital, National University Health System, Singapore, Singapore.
  • Department of Emergency Medicine, Indiana University School of Medicine, Indianapolis, IN, United States.

ORCID profiles: Jeffrey Micheal Franc, Attila Julius Hertelendy, Lenard Cheng, Ryan Hata, Manuela Verde

Cite

APA

Franc, J. M., Hertelendy, A. J., Cheng, L., Hata, R., & Verde, M. (2024). Accuracy of a Commercial Large Language Model (ChatGPT) to Perform Disaster Triage of Simulated Patients Using the Simple Triage and Rapid Treatment (START) Protocol: Gage Repeatability and Reproducibility Study. Journal of Medical Internet Research, 26, e55648. https://doi.org/10.2196/55648

Vancouver

Franc JM, Hertelendy AJ, Cheng L, Hata R, Verde M. Accuracy of a Commercial Large Language Model (ChatGPT) to Perform Disaster Triage of Simulated Patients Using the Simple Triage and Rapid Treatment (START) Protocol: Gage Repeatability and Reproducibility Study. Journal of Medical Internet Research. 2024;26:e55648. doi:10.2196/55648

BibTeX
@article{Franc2024,
  author = {Franc, Jeffrey Micheal and Hertelendy, Attila Julius and Cheng, Lenard and Hata, Ryan and Verde, Manuela},
  title = {Accuracy of a Commercial Large Language Model (ChatGPT) to Perform Disaster Triage of Simulated Patients Using the Simple Triage and Rapid Treatment (START) Protocol: Gage Repeatability and Reproducibility Study},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  volume = {26},
  pages = {e55648},
  doi = {10.2196/55648}
}
Start for free →