Leveraging large language models for generating responses to patient messages—a subjective analysis
Journal of the American Medical Informatics Association · 20.05.2024 · 31 (6) · 1367-1379
DOI: 10.1093/jamia/ocae052 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 18.03.2024. Verfasst von Siru Liu und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37212, United States. Hauptthemen laut MEDLINE: Patient Portals. Die Arbeit wurde bisher 84-mal zitiert (Crossref) und nennt 34 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NLM NIH HHS, NIA NIH HHS, NIH HHS.
Abstract (Originalsprache)
Objective
This study aimed to develop and assess the performance of fine-tuned large language models for generating responses to patient messages sent via an electronic health record patient portal. Materials and Methods Utilizing a dataset of messages and responses extracted from the patient portal at a large academic medical center, we developed a model (CLAIR-Short) based on a pre-trained large language model (LLaMA-65B). In addition, we used the OpenAI API to update physician responses from an open-source dataset into a format with informative paragraphs that offered patient education while emphasizing empathy and professionalism. By combining with this dataset, we further fine-tuned our model (CLAIR-Long). To evaluate fine-tuned models, we used 10 representative patient portal questions in primary care to generate responses. We asked primary care physicians to review generated responses from our models and ChatGPT and rated them for empathy, responsiveness, accuracy, and usefulness.
Results
The dataset consisted of 499 794 pairs of patient messages and corresponding responses from the patient portal, with 5000 patient messages and ChatGPT-updated responses from an online platform. Four primary care physicians participated in the survey. CLAIR-Short exhibited the ability to generate concise responses similar to provider’s responses. CLAIR-Long responses provided increased patient educational content compared to CLAIR-Short and were rated similarly to ChatGPT’s responses, receiving positive evaluations for responsiveness, empathy, and accuracy, while receiving a neutral rating for usefulness.
Conclusion
This subjective analysis suggests that leveraging large language models to generate responses to patient messages demonstrates significant potential in facilitating communication between patients and healthcare providers.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37212, United States.
- Department of Medicine, Vanderbilt University Medical Center, Nashville, TN 37212, United States.
- Department of General Internal Medicine and Public Health, Vanderbilt University Medical Center, Nashville, TN 37212, United States.
- Department of Medicine, Stanford University, Stanford, CA 94304, United States.
ORCID-Profile: Allison B McCoy, Adam Wright
Förderung
- NLM NIH HHS
- NIA NIH HHS
- NIH HHS
- NIH
Zitieren
APA
Liu, S., McCoy, A. B., Wright, A. P., Carew, B., Genkins, J. Z., Huang, S. S., Peterson, J. F., Steitz, B., & Wright, A. (2024). Leveraging large language models for generating responses to patient messages—a subjective analysis. Journal of the American Medical Informatics Association, 31(6), 1367-1379. https://doi.org/10.1093/jamia/ocae052
Vancouver
Liu S, McCoy AB, Wright AP, Carew B, Genkins JZ, Huang SS, et al. Leveraging large language models for generating responses to patient messages—a subjective analysis. Journal of the American Medical Informatics Association. 2024;31(6):1367-1379. doi:10.1093/jamia/ocae052
BibTeX
@article{Liu2024,
author = {Liu, Siru and McCoy, Allison B and Wright, Aileen P and Carew, Babatunde and Genkins, Julian Z and Huang, Sean S and Peterson, Josh F and Steitz, Bryan and Wright, Adam},
title = {Leveraging large language models for generating responses to patient messages—a subjective analysis},
journal = {Journal of the American Medical Informatics Association},
year = {2024},
volume = {31},
number = {6},
pages = {1367--1379},
doi = {10.1093/jamia/ocae052}
}
EUVIMED