EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Prompt engineering on leveraging large language models in generating response to InBasket messages

Sherry Yan, Wendi Knapp, Andrew Leong, Sarira Kadkhodazadeh, Souvik Das, Veena G Jones, Robert Clark, David Grattendick, Kevin Chen, Lisa Hladik, Lawrence Fagan, Albert Chan

Journal of the American Medical Informatics Association · 01.10.2024 · 31 (10) · 2263-2270

DOI: 10.1093/jamia/ocae172 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 19.07.2024. Verfasst von Sherry Yan und 11 weiteren Autor:innen. Beteiligte Einrichtung: Center for Health Systems Research, Sutter Health, Walnut Creek, CA 94596, United States. Hauptthemen laut MEDLINE: Electronic Health Records. Die Arbeit wurde bisher 48-mal zitiert (Crossref) und nennt 24 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

Large Language Models (LLMs) have been proposed as a solution to address high volumes of Patient Medical Advice Requests (PMARs). This study addresses whether LLMs can generate high quality draft responses to PMARs that satisfies both patients and clinicians with prompt engineering. Materials and Methods We designed a novel human-involved iterative processes to train and validate prompts to LLM in creating appropriate responses to PMARs. GPT-4 was used to generate response to the messages. We updated the prompts, and evaluated both clinician and patient acceptance of LLM-generated draft responses at each iteration, and tested the optimized prompt on independent validation data sets. The optimized prompt was implemented in the electronic health record production environment and tested by 69 primary care clinicians.

Results

After 3 iterations of prompt engineering, physician acceptance of draft suitability increased from 62% to 84% (P <.001) in the validation dataset (N = 200), and 74% of drafts in the test dataset were rated as “helpful.” Patients also noted significantly increased favorability of message tone (78%) and overall quality (80%) for the optimized prompt compared to the original prompt in the training dataset, patients were unable to differentiate human and LLM-generated draft PMAR responses for 76% of the messages, in contrast to the earlier preference for human-generated responses. Majority (72%) of clinicians believed it can reduce cognitive load in dealing with InBasket messages. Discussion and Conclusion Informed by clinician and patient feedback synergistically, tuning in LLM prompt alone can be effective in creating clinically relevant and useful draft responses to PMARs.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Electronic Health RecordsHumansNatural Language ProcessingReminder Systems

Stichwörter der Autor:innen

ChatGPTInBasket messagepatient and clinician perspetivepatient medical advice requestprompt engineering

Autor:innen und Institutionen

  • Center for Health Systems Research, Sutter Health, Walnut Creek, CA 94596, United States.
  • Sutter Health Data Science Team, Sutter Health, Sacramento, CA 95833, United States.
  • Palo Alto Foundation Medical Group, Sutter Health, Palo Alto, CA 94301, United States.
  • Sutter Digital Engagement, Sutter Health, Sacramento, CA 95833, United States.
  • Sutter Data and Analytics Department, Sutter Health, Sacramento, CA 95833, United States.
  • Sutter Medical Group, Sutter Health, Roseville, CA 95661, United States.

Zitieren

APA

Yan, S., Knapp, W., Leong, A., Kadkhodazadeh, S., Das, S., Jones, V. G., Clark, R., Grattendick, D., Chen, K., Hladik, L., Fagan, L., & Chan, A. (2024). Prompt engineering on leveraging large language models in generating response to InBasket messages. Journal of the American Medical Informatics Association, 31(10), 2263-2270. https://doi.org/10.1093/jamia/ocae172

Vancouver

Yan S, Knapp W, Leong A, Kadkhodazadeh S, Das S, Jones VG, et al. Prompt engineering on leveraging large language models in generating response to InBasket messages. Journal of the American Medical Informatics Association. 2024;31(10):2263-2270. doi:10.1093/jamia/ocae172

BibTeX
@article{Yan2024,
  author = {Yan, Sherry and Knapp, Wendi and Leong, Andrew and Kadkhodazadeh, Sarira and Das, Souvik and Jones, Veena G and Clark, Robert and Grattendick, David and Chen, Kevin and Hladik, Lisa and Fagan, Lawrence and Chan, Albert},
  title = {Prompt engineering on leveraging large language models in generating response to InBasket messages},
  journal = {Journal of the American Medical Informatics Association},
  year = {2024},
  volume = {31},
  number = {10},
  pages = {2263--2270},
  doi = {10.1093/jamia/ocae172}
}
Kostenlos starten →