EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access

Triage Performance Across Large Language Models, ChatGPT, and Untrained Doctors in Emergency Medicine: Comparative Study

Lars Masanneck iD, Linea Schmidt iD, Antonia Seifert iD, Tristan Kölsche iD, Niklas Huntemann iD, Robin Jansen iD, Mohammed Mehsin iD, Michael Bernhard iD, Sven G Meuth iD, Lennert Böhm iD, Marc Pawlitzki iD

Journal of Medical Internet Research · 14.06.2024 · 26 · e53297

DOI: 10.2196/53297 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 14.06.2024. Verfasst von Lars Masanneck und 10 weiteren Autor:innen. Beteiligte Einrichtung: Department of Neurology, Medical Faculty and University Hospital Düsseldorf, Heinrich Heine University Düsseldorf, Düsseldorf, Germany. Hauptthemen laut MEDLINE: Triage, Emergency Medicine. Die Arbeit wurde bisher 144-mal zitiert (Crossref) und nennt 41 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models (LLMs) have demonstrated impressive performances in various medical domains, prompting an exploration of their potential utility within the high-demand setting of emergency department (ED) triage. This study evaluated the triage proficiency of different LLMs and ChatGPT, an LLM-based chatbot, compared to professionally trained ED staff and untrained personnel. We further explored whether LLM responses could guide untrained staff in effective triage.

Objective

This study aimed to assess the efficacy of LLMs and the associated product ChatGPT in ED triage compared to personnel of varying training status and to investigate if the models’ responses can enhance the triage proficiency of untrained personnel.

Methods

A total of 124 anonymized case vignettes were triaged by untrained doctors; different versions of currently available LLMs; ChatGPT; and professionally trained raters, who subsequently agreed on a consensus set according to the Manchester Triage System (MTS). The prototypical vignettes were adapted from cases at a tertiary ED in Germany. The main outcome was the level of agreement between raters’ MTS level assignments, measured via quadratic-weighted Cohen κ. The extent of over- and undertriage was also determined. Notably, instances of ChatGPT were prompted using zero-shot approaches without extensive background information on the MTS. The tested LLMs included raw GPT-4, Llama 3 70B, Gemini 1.5, and Mixtral 8x7b.

Results

GPT-4–based ChatGPT and untrained doctors showed substantial agreement with the consensus triage of professional raters (κ=mean 0.67, SD 0.037 and κ=mean 0.68, SD 0.056, respectively), significantly exceeding the performance of GPT-3.5–based ChatGPT (κ=mean 0.54, SD 0.024; P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

TriageEmergency MedicineHumansPhysiciansEmergency Service, HospitalLanguageGermanyFemale

Stichwörter der Autor:innen

ChatGPTGermanGermanyartificial intelligencecohortcomparative studydigital healthdoctordoctorsemergency medicinelarge language modelspersonnelstafftriageuntrained doctors

Autor:innen und Institutionen

  • Department of Neurology, Medical Faculty and University Hospital Düsseldorf, Heinrich Heine University Düsseldorf, Düsseldorf, Germany.
  • Digital Health Center, Hasso Plattner Institute, University of Potsdam, Potsdam, Germany.
  • Emergency Department, Medical Faculty and University Hospital Düsseldorf, Heinrich Heine University Düsseldorf, Düsseldorf, Germany.

ORCID-Profile: Lars Masanneck, Linea Schmidt, Antonia Seifert, Tristan Kölsche, Niklas Huntemann, Robin Jansen, Mohammed Mehsin, Michael Bernhard, Sven G Meuth, Lennert Böhm, Marc Pawlitzki

Zitieren

APA

Masanneck, L., Schmidt, L., Seifert, A., Kölsche, T., Huntemann, N., Jansen, R., Mehsin, M., Bernhard, M., Meuth, S. G., Böhm, L., & Pawlitzki, M. (2024). Triage Performance Across Large Language Models, ChatGPT, and Untrained Doctors in Emergency Medicine: Comparative Study. Journal of Medical Internet Research, 26, e53297. https://doi.org/10.2196/53297

Vancouver

Masanneck L, Schmidt L, Seifert A, Kölsche T, Huntemann N, Jansen R, et al. Triage Performance Across Large Language Models, ChatGPT, and Untrained Doctors in Emergency Medicine: Comparative Study. Journal of Medical Internet Research. 2024;26:e53297. doi:10.2196/53297

BibTeX
@article{Masanneck2024,
  author = {Masanneck, Lars and Schmidt, Linea and Seifert, Antonia and Kölsche, Tristan and Huntemann, Niklas and Jansen, Robin and Mehsin, Mohammed and Bernhard, Michael and Meuth, Sven G and Böhm, Lennert and Pawlitzki, Marc},
  title = {Triage Performance Across Large Language Models, ChatGPT, and Untrained Doctors in Emergency Medicine: Comparative Study},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  volume = {26},
  pages = {e53297},
  doi = {10.2196/53297}
}
Kostenlos starten →