Prompt engineering on leveraging large language models in generating response to InBasket messages
Journal of the American Medical Informatics Association · 01.10.2024 · 31 (10) · 2263-2270
DOI: 10.1093/jamia/ocae172 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 19.07.2024. Verfasst von Sherry Yan und 11 weiteren Autor:innen. Beteiligte Einrichtung: Center for Health Systems Research, Sutter Health, Walnut Creek, CA 94596, United States. Hauptthemen laut MEDLINE: Electronic Health Records. Die Arbeit wurde bisher 48-mal zitiert (Crossref) und nennt 24 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).
Abstract (Originalsprache)
Objectives
Large Language Models (LLMs) have been proposed as a solution to address high volumes of Patient Medical Advice Requests (PMARs). This study addresses whether LLMs can generate high quality draft responses to PMARs that satisfies both patients and clinicians with prompt engineering. Materials and Methods We designed a novel human-involved iterative processes to train and validate prompts to LLM in creating appropriate responses to PMARs. GPT-4 was used to generate response to the messages. We updated the prompts, and evaluated both clinician and patient acceptance of LLM-generated draft responses at each iteration, and tested the optimized prompt on independent validation data sets. The optimized prompt was implemented in the electronic health record production environment and tested by 69 primary care clinicians.
Results
After 3 iterations of prompt engineering, physician acceptance of draft suitability increased from 62% to 84% (P <.001) in the validation dataset (N = 200), and 74% of drafts in the test dataset were rated as “helpful.” Patients also noted significantly increased favorability of message tone (78%) and overall quality (80%) for the optimized prompt compared to the original prompt in the training dataset, patients were unable to differentiate human and LLM-generated draft PMAR responses for 76% of the messages, in contrast to the earlier preference for human-generated responses. Majority (72%) of clinicians believed it can reduce cognitive load in dealing with InBasket messages. Discussion and Conclusion Informed by clinician and patient feedback synergistically, tuning in LLM prompt alone can be effective in creating clinically relevant and useful draft responses to PMARs.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Center for Health Systems Research, Sutter Health, Walnut Creek, CA 94596, United States.
- Sutter Health Data Science Team, Sutter Health, Sacramento, CA 95833, United States.
- Palo Alto Foundation Medical Group, Sutter Health, Palo Alto, CA 94301, United States.
- Sutter Digital Engagement, Sutter Health, Sacramento, CA 95833, United States.
- Sutter Data and Analytics Department, Sutter Health, Sacramento, CA 95833, United States.
- Sutter Medical Group, Sutter Health, Roseville, CA 95661, United States.
Zitieren
APA
Yan, S., Knapp, W., Leong, A., Kadkhodazadeh, S., Das, S., Jones, V. G., Clark, R., Grattendick, D., Chen, K., Hladik, L., Fagan, L., & Chan, A. (2024). Prompt engineering on leveraging large language models in generating response to InBasket messages. Journal of the American Medical Informatics Association, 31(10), 2263-2270. https://doi.org/10.1093/jamia/ocae172
Vancouver
Yan S, Knapp W, Leong A, Kadkhodazadeh S, Das S, Jones VG, et al. Prompt engineering on leveraging large language models in generating response to InBasket messages. Journal of the American Medical Informatics Association. 2024;31(10):2263-2270. doi:10.1093/jamia/ocae172
BibTeX
@article{Yan2024,
author = {Yan, Sherry and Knapp, Wendi and Leong, Andrew and Kadkhodazadeh, Sarira and Das, Souvik and Jones, Veena G and Clark, Robert and Grattendick, David and Chen, Kevin and Hladik, Lisa and Fagan, Lawrence and Chan, Albert},
title = {Prompt engineering on leveraging large language models in generating response to InBasket messages},
journal = {Journal of the American Medical Informatics Association},
year = {2024},
volume = {31},
number = {10},
pages = {2263--2270},
doi = {10.1093/jamia/ocae172}
}
EUVIMED