Developing and Evaluating Large Language Model–Generated Emergency Medicine Handoff Notes
Entwicklung und Bewertung von Large-Language-Model-generierten Übergabevermerken in der Notfallmedizin
JAMA Network Open · 03.12.2024 · 7 (12) · e2448723
DOI: 10.1001/jamanetworkopen.2024.48723 ↗Auf einen Blick
LLM-generierte EM-zu-IP-Übergabevermerke zeigten in 1600 Patient:innen mehr Ähnlichkeit zu den Arzttexten als diese selbst. In einer Sicherheitsbewertung waren sie jedoch etwas schlechter als ärztliche Notizen.
Fragestellung
Wie genau und sicher sind von einem Large Language Model erzeugte EM-zu-IP-Übergabevermerke im Vergleich zu ärztlich verfassten Notizen?
Methode
Kohortenstudie mit 1600 EM-Patientenakten; Vergleich von LLM-generierten mit ärztlichen Übergabevermerken anhand von ROUGE, BERTScore, SCALE und einer neuen Sicherheitsbewertung. Analyse von Oktober 2023 bis März 2024.
Ergebnisse
1600 EM-Patientenakten, 832 Frauen, mittleres Alter 59,9 Jahre. LLM-Notizen: ROUGE 0,322 vs. 0,088, BERTScore 0,859 vs. 0,796, SCALE 0,691 vs. 0,456. Nutzen 4,04 vs. 4,36; Sicherheit 4,06 vs. 4,50.
Schlussfolgerung der Autor:innen
Die Autor:innen sehen die LLM-Notizen nach konventionellen automatischen Verfahren als überlegen, nach der neuen Bewertungsmethode aber als leicht schlechter in Nutzen und Sicherheit. Sie betonen ein Physician-in-the-loop-Design.
Kernaussagen
- LLM-Notizen hatten höhere ROUGE-, BERTScore- und SCALE-Werte
- Nutzen- und Sicherheitswerte waren etwas niedriger als bei Arzttexten
- Keine LLM-Notiz wurde als kritisches Patientensicherheitsrisiko eingestuft
Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.
Kurz gefasst
Fachartikel in der Zeitschrift JAMA Network Open, online erschienen am 03.12.2024. Verfasst von Vince Hartman und 8 weiteren Autor:innen. Beteiligte Einrichtung: Abstractive Health, New York, New York. Hauptthemen laut MEDLINE: Patient Handoff, Electronic Health Records, Emergency Medicine. Die Arbeit wurde bisher 85-mal zitiert (Crossref) und nennt 56 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIA NIH HHS, NCATS NIH HHS.
Abstract (Originalsprache)
Importance An emergency medicine (EM) handoff note generated by a large language model (LLM) has the potential to reduce physician documentation burden without compromising the safety of EM-to-inpatient (IP) handoffs.
Objective
To develop LLM-generated EM-to-IP handoff notes and evaluate their accuracy and safety compared with physician-written notes. Design, Setting, and Participants This cohort study used EM patient medical records with acute hospital admissions that occurred in 2023 at NewYork-Presbyterian/Weill Cornell Medical Center. A customized clinical LLM pipeline was trained, tested, and evaluated to generate templated EM-to-IP handoff notes. Using both conventional automated methods (ie, recall-oriented understudy for gisting evaluation [ROUGE], bidirectional encoder representations from transformers score [BERTScore], and source chunking approach for large-scale inconsistency evaluation [SCALE]) and a novel patient safety-focused framework, LLM-generated handoff notes vs physician-written notes were compared. Data were analyzed from October 2023 to March 2024. Exposure LLM-generated EM handoff notes. Main Outcomes and Measures LLM-generated handoff notes were evaluated for (1) lexical similarity with respect to physician-written notes using ROUGE and BERTScore; (2) fidelity with respect to source notes using SCALE; and (3) readability, completeness, curation, correctness, usefulness, and implications for patient safety using a novel framework.
Results
In this study of 1600 EM patient records (832 [52%] female and mean [SD] age of 59.9 [18.9] years), LLM-generated handoff notes, compared with physician-written ones, had higher ROUGE (0.322 vs 0.088), BERTScore (0.859 vs 0.796), and SCALE scores (0.691 vs 0.456), indicating the LLM-generated summaries exhibited greater similarity and more detail. As reviewed by 3 board-certified EM physicians, a subsample of 50 LLM-generated summaries had a mean (SD) usefulness score of 4.04 (0.86) out of 5 (compared with 4.36 [0.71] for physician-written) and mean (SD) patient safety scores of 4.06 (0.86) out of 5 (compared with 4.50 [0.56] for physician-written). None of the LLM-generated summaries were classified as a critical patient safety risk. Conclusions and Relevance In this cohort study of 1600 EM patient medical records, LLM-generated EM-to-IP handoff notes were determined superior compared with physician-written summaries via conventional automated evaluation methods, but marginally inferior in usefulness and safety via a novel evaluation framework. This study suggests the importance of a physician-in-loop implementation design for this model and demonstrates an effective strategy to measure preimplementation patient safety of LLM models.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Abstractive Health, New York, New York.
- Department of Emergency Medicine, NewYork-Presbyterian/Weill Cornell Medicine, New York.
- Department of Population Health, NewYork-Presbyterian/Weill Cornell Medicine, New York.
- Clinical and Translational Science Center, Weill Cornell Medicine, New York, New York.
Förderung
- NIA NIH HHS
- NCATS NIH HHS
Zitieren
APA
Hartman, V., Zhang, X., Poddar, R., McCarty, M., Fortenko, A., Sholle, E., Sharma, R., Campion, T., & Steel, P. A. D. (2024). Developing and Evaluating Large Language Model–Generated Emergency Medicine Handoff Notes. JAMA Network Open, 7(12), e2448723. https://doi.org/10.1001/jamanetworkopen.2024.48723
Vancouver
Hartman V, Zhang X, Poddar R, McCarty M, Fortenko A, Sholle E, et al. Developing and Evaluating Large Language Model–Generated Emergency Medicine Handoff Notes. JAMA Network Open. 2024;7(12):e2448723. doi:10.1001/jamanetworkopen.2024.48723
BibTeX
@article{Hartman2024,
author = {Hartman, Vince and Zhang, Xinyuan and Poddar, Ritika and McCarty, Matthew and Fortenko, Alexander and Sholle, Evan and Sharma, Rahul and Campion, Thomas and Steel, Peter A D},
title = {Developing and Evaluating Large Language Model–Generated Emergency Medicine Handoff Notes},
journal = {JAMA Network Open},
year = {2024},
volume = {7},
number = {12},
pages = {e2448723},
doi = {10.1001/jamanetworkopen.2024.48723}
}
EUVIMED