Danger, Danger, Gaston Labat! Does zero-shot artificial intelligence correlate with anticoagulation guidelines recommendations for neuraxial anesthesia?
Regional Anesthesia & Pain Medicine · 2024-09 · 49 (9) · 661-667
DOI: 10.1136/rapm-2023-104868 ↗Kurz gefasst
Leitlinie in der Zeitschrift Regional Anesthesia & Pain Medicine, online erschienen am 22.01.2024. Verfasst von Nathan C Hurley und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Anesthesiology, University of Wisconsin-Madison, Madison, Wisconsin, USA. Hauptthemen laut MEDLINE: Anticoagulants, Artificial Intelligence, Anesthesia, Conduction, Practice Guidelines as Topic. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 32 Literaturangaben.
Abstract (Originalsprache)
Introduction
Artificial intelligence and large language models (LLMs) have emerged as potentially disruptive technologies in healthcare. In this study GPT-3.5, an accessible LLM, was assessed for its accuracy and reliability in performing guideline-based evaluation of neuraxial bleeding risk in hypothetical patients on anticoagulation medication. The study also explored the impact of structured prompt guidance on the LLM’s performance.
Methods
A dataset of 10 hypothetical patient stems and 26 anticoagulation profiles (260 unique combinations) was developed based on American Society of Regional Anesthesia and Pain Medicine guidelines. Five prompts were created for the LLM, ranging from minimal guidance to explicit instructions. The model’s responses were compared with a “truth table” based on the guidelines. Performance metrics, including accuracy and area under the receiver operating curve (AUC), were used.
Results
Baseline performance of GPT-3.5 was slightly above chance. With detailed prompts and explicit guidelines, performance improved significantly (AUC 0.70, 95% CI (0.64 to 0.77)). Performance varied among medication classes.
Discussion
LLMs show potential for assisting in clinical decision making but rely on accurate and relevant prompts. Integration of LLMs should consider safety and privacy concerns. Further research is needed to optimize LLM performance and address complex scenarios. The tested LLM demonstrates potential in assessing neuraxial bleeding risk but relies on precise prompts. LLM integration should be approached cautiously, considering limitations. Future research should focus on optimization and understanding LLM capabilities and limitations in healthcare.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Substanzen
Autor:innen und Institutionen
- Department of Anesthesiology, University of Wisconsin-Madison, Madison, Wisconsin, USA.
- Anesthesiology, Vanderbilt University Medical Center, Nashville, Tennessee, USA.
- Anesthesiology, University of Wisconsin-Madison, Madison, Wisconsin, USA.
- Department of Anesthesiology, University of Wisconsin-Madison, Madison, Wisconsin, USA ahess5@wisc.edu.
- Department of Pathology and Laboratory Medicine, University of Wisconsin-Madison, Madison, Wisconsin, USA.
ORCID-Profile: Nathan C Hurley, Rajnish K Gupta
Zitieren
APA
Hurley, N. C., Gupta, R. K., Schroeder, K. M., & Hess, A. S. (2024). Danger, Danger, Gaston Labat! Does zero-shot artificial intelligence correlate with anticoagulation guidelines recommendations for neuraxial anesthesia?. Regional Anesthesia & Pain Medicine, 49(9), 661-667. https://doi.org/10.1136/rapm-2023-104868
Vancouver
Hurley NC, Gupta RK, Schroeder KM, Hess AS. Danger, Danger, Gaston Labat! Does zero-shot artificial intelligence correlate with anticoagulation guidelines recommendations for neuraxial anesthesia?. Regional Anesthesia & Pain Medicine. 2024;49(9):661-667. doi:10.1136/rapm-2023-104868
BibTeX
@article{Hurley2024,
author = {Hurley, Nathan C and Gupta, Rajnish K and Schroeder, Kristopher M and Hess, Aaron S},
title = {Danger, Danger, Gaston Labat! Does zero-shot artificial intelligence correlate with anticoagulation guidelines recommendations for neuraxial anesthesia?},
journal = {Regional Anesthesia & Pain Medicine},
year = {2024},
volume = {49},
number = {9},
pages = {661--667},
doi = {10.1136/rapm-2023-104868}
}
EUVIMED