EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

A proof-of-concept study for patient use of open notes with large language models

Liz Salmi iD, Dana M Lewis iD, Jennifer L Clarke iD, Zhiyong Dong iD, Rudy Fischmann iD, Emily I McIntosh iD, Chethan R Sarabu iD, Catherine M DesRoches iD

JAMIA Open · 06.03.2025 · 8 (2)

DOI: 10.1093/jamiaopen/ooaf021 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JAMIA Open, online erschienen am 09.04.2025. Verfasst von Liz Salmi und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Women's and Children's Health, Uppsala University, 752 37 Uppsala, Sweden. Die Arbeit wurde bisher 26-mal zitiert (Crossref) und nennt 46 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Patrick J. McGovern Foundation.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

The use of large language models (LLMs) is growing for both clinicians and patients. While researchers and clinicians have explored LLMs to manage patient portal messages and reduce burnout, there is less documentation about how patients use these tools to understand clinical notes and inform decision-making. This proof-of-concept study examined the reliability and accuracy of LLMs in responding to patient queries based on an open visit note. Materials and Methods In a cross-sectional proof-of-concept study, 3 commercially available LLMs (ChatGPT 4o, Claude 3 Opus, Gemini 1.5) were evaluated using 4 distinct prompt series—Standard, Randomized, Persona, and Randomized Persona—with multiple questions, designed by patients, in response to a single neuro-oncology progress note. LLM responses were scored by the note author (neuro-oncologist) and a patient who receives care from the note author, using an 8-criterion rubric that assessed Accuracy, Relevance, Clarity, Actionability, Empathy/Tone, Completeness, Evidence, and Consistency. Descriptive statistics were used to summarize the performance of each LLM across all prompts.

Results

Overall, the Standard and Persona-based prompt series yielded the best results across all criterion regardless of LLM. Chat-GPT 4o using Persona-based prompts scored highest in all categories. All LLMs scored low in the use of Evidence.

Discussion

This proof-of-concept study highlighted the potential for LLMs to assist patients in interpreting open notes. The most effective LLM responses were achieved by applying Persona-style prompts to a patient’s question.

Conclusion

Optimizing LLMs for patient-driven queries, and patient education and counseling around the use of LLMs, have potential to enhance patient use and understanding of their health information.

Schlagwörter

Stichwörter der Autor:innen

generative AIlarge language modelsopen notespatient portals

Autor:innen und Institutionen

  • Department of Women's and Children's Health, Uppsala University, 752 37 Uppsala, Sweden.
  • OpenNotes, Beth Israel Deaconess Medical Center, Boston, MA 02215, United States.
  • #OpenAPS, Seattle, WA 98101, United States.
  • Department of Neurological Surgery, University of California, San Francisco, CA 94117, United States.
  • Richmond, VA 23120, United States.
  • Guelph, ON N1E 5J5, Canada.

ORCID-Profile: Liz Salmi, Dana M Lewis, Jennifer L Clarke, Zhiyong Dong, Rudy Fischmann, Emily I McIntosh, Chethan R Sarabu, Catherine M DesRoches

Förderung

  • Patrick J. McGovern Foundation

Zitieren

APA

Salmi, L., Lewis, D. M., Clarke, J. L., Dong, Z., Fischmann, R., McIntosh, E. I., Sarabu, C. R., & DesRoches, C. M. (2025). A proof-of-concept study for patient use of open notes with large language models. JAMIA Open, 8(2). https://doi.org/10.1093/jamiaopen/ooaf021

Vancouver

Salmi L, Lewis DM, Clarke JL, Dong Z, Fischmann R, McIntosh EI, et al. A proof-of-concept study for patient use of open notes with large language models. JAMIA Open. 2025;8(2). doi:10.1093/jamiaopen/ooaf021

BibTeX
@article{Salmi2025,
  author = {Salmi, Liz and Lewis, Dana M and Clarke, Jennifer L and Dong, Zhiyong and Fischmann, Rudy and McIntosh, Emily I and Sarabu, Chethan R and DesRoches, Catherine M},
  title = {A proof-of-concept study for patient use of open notes with large language models},
  journal = {JAMIA Open},
  year = {2025},
  volume = {8},
  number = {2},
  doi = {10.1093/jamiaopen/ooaf021}
}
Kostenlos starten →