Detecting emergencies in patient portal messages using large language models and knowledge graph-based retrieval-augmented generation
Journal of the American Medical Informatics Association · 01.06.2025 · 32 (6) · 1032-1039
DOI: 10.1093/jamia/ocaf059 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 12.04.2025. Verfasst von Siru Liu und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37212, United States. Hauptthemen laut MEDLINE: Triage, Patient Portals, Emergencies, Information Storage and Retrieval. Die Arbeit wurde bisher 52-mal zitiert (Crossref) und nennt 44 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: NLM NIH HHS, NIH HHS, NIA NIH HHS.
Abstract (Originalsprache)
Objectives
This study aims to develop and evaluate an approach using large language models (LLMs) and a knowledge graph to triage patient messages that need emergency care. The goal is to notify patients when their messages indicate an emergency, guiding them to seek immediate help rather than using the patient portal, to improve patient safety. Materials and Methods We selected 1020 messages sent to Vanderbilt University Medical Center providers between January 1, 2022 and March 7, 2023. We developed four models to triage these messages for emergencies: (1) Prompt-Only: the patient message was input with a prompt directly into the LLM; (2) Naïve Retrieval Augmented Generation (RAG): provided retrieved information as context to the LLM; (3) RAG from Knowledge Graph with Local Search: a knowledge graph was used to retrieve locally relevant information based on semantic similarities; (4) RAG from Knowledge Graph with Global Search: a knowledge graph was used to retrieve globally relevant information through hierarchical community detection. The knowledge base was a triage book covering 225 protocols.
Results
The RAG from Knowledge Graph model with global search outperformed other models, achieving an accuracy of 0.99, a sensitivity of 0.98, and a specificity of 0.99. It demonstrated significant improvements in triaging emergency messages compared to LLM without RAG and naïve RAG.
Discussion
The traditional LLM without any retrieval mechanism underperformed compared to models with RAG, which aligns with the expected benefits of augmenting LLMs with domain-specific knowledge sources. Our results suggest that providing external knowledge, especially in a structured manner and in community summaries, can improve LLM performance in triaging patient portal messages.
Conclusion
LLMs can effectively assist in triaging emergency patient messages after integrating with a knowledge graph about a nurse triage book. Future research should focus on expanding the knowledge graph and deploying the system to evaluate its impact on patient outcomes.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37212, United States.
- Department of Computer Science, Vanderbilt University, Nashville, TN 37240, United States.
- Department of Medicine, Vanderbilt University Medical Center, Nashville, TN 37232, United States.
ORCID-Profile: Allison B McCoy, Adam Wright
Förderung
- NLM NIH HHS
- NIH HHS
- NIA NIH HHS
- NIH
Zitieren
APA
Liu, S., Wright, A. P., McCoy, A. B., Huang, S. S., Steitz, B., & Wright, A. (2025). Detecting emergencies in patient portal messages using large language models and knowledge graph-based retrieval-augmented generation. Journal of the American Medical Informatics Association, 32(6), 1032-1039. https://doi.org/10.1093/jamia/ocaf059
Vancouver
Liu S, Wright AP, McCoy AB, Huang SS, Steitz B, Wright A. Detecting emergencies in patient portal messages using large language models and knowledge graph-based retrieval-augmented generation. Journal of the American Medical Informatics Association. 2025;32(6):1032-1039. doi:10.1093/jamia/ocaf059
BibTeX
@article{Liu2025,
author = {Liu, Siru and Wright, Aileen P and McCoy, Allison B and Huang, Sean S and Steitz, Bryan and Wright, Adam},
title = {Detecting emergencies in patient portal messages using large language models and knowledge graph-based retrieval-augmented generation},
journal = {Journal of the American Medical Informatics Association},
year = {2025},
volume = {32},
number = {6},
pages = {1032--1039},
doi = {10.1093/jamia/ocaf059}
}
EUVIMED