EUVIMEDAvasta EUVIMED
Open access

Large Language Models’ Simulation of Patient Comprehension of Discharge Summaries: Multimodel Evaluation Study

Simulation des Patientenverständnisses von Entlassungsbriefen durch Large Language Models: Multimodell-Evaluationsstudie

Xinyao Ma, Rui Zhu, Zihao Wang, Jingwei Xiong, Qingyu Chen, L Jean Camp, Haixu Tang

Journal of Medical Internet Research · 2026-10-22 · 28 · e89003-e89003

DOI: 10.2196/89003 ↗

Auf einen Blick

Vier LLMs wurden mit 99 Prolific-Teilnehmenden verglichen, um Antworten zu Entlassungsbriefen zu simulieren. Die Übereinstimmung war bei wissensbasierten Items besser als bei Wahrnehmungsitems.

Fragestellung

Können aktuelle LLMs unterschiedliche Patientenperspektiven zu Entlassungsbriefen per Persona-Prompting abbilden, und ist dies über Subgruppen hinweg gleichmäßig?

Methode

Vergleich von 4 LLMs mit Antworten von 99 Personen; 10 Fragen zu Entlassungsbriefen, Auswertung mit modal mean absolute error und expected absolute ordinal distance.

Ergebnisse

Die Übereinstimmung war bei informationsbasierten Items höher als bei Wahrnehmungsitems. Alle Modelle passten bei Informationsitems besser zur Hochbildungs- als zur Niedrigbildungsgruppe; Kurzfassungen mit klinischer Komplexität zeigten die größte Abweichung.

Schlussfolgerung der Autor:innen

LLMs können häufige Antwortmuster und bildungsbezogene Personas annähern, erfassen heterogene und wahrnehmungsbezogene Antworten aber weniger zuverlässig. Die Modellwahl beeinflusst die Fairness deutlich.

Kernaussagen

  • Bessere Übereinstimmung bei faktenbasierten als bei Wahrnehmungs-Items
  • Hochbildungsgruppe wurde konsistenter abgebildet als Niedrigbildungsgruppe
  • „I don’t know“ und „Not provided“ wurden von LLMs seltener gewählt

Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.

At a glance

Journal article in Journal of Medical Internet Research, published online on 2026-10-22. Written by Xinyao Ma and 6 co-authors. Institution: Luddy School of Informatics, Computing, and Engineering, Indiana University Bloomington, 700 N Woodlawn Ave, Bloomington, IN, United States, 1 812-22…. Major MEDLINE topics: Large Language Models, Comprehension, Patient Discharge, Patient Discharge Summaries. 56 references. The full text is freely available in PubMed Central.

Auf einen BlickAbstraktSubjectsBibliograafilised andmedCiteRelated studies

Abstract (original language)

Background

Large language models (LLMs) are increasingly used to generate and evaluate patient-facing medical materials. Through persona prompting, LLMs can be instructed to simulate responses from individuals with different demographic and experiential backgrounds, offering a scalable approach for approximating patient feedback. However, it remains unclear how closely these simulated persona-based responses reflect real patient perspectives, and whether this alignment is consistent across demographic subgroups.

Objective

This study assessed whether contemporary LLMs can simulate diverse patient perspectives on hospital discharge summaries when prompted with personas defined by education, gender, and health care use. We further examined whether persona-based simulation performance was equitable across subgroups by comparing LLM-generated responses with responses from a general-population human cohort under a standardized evaluation framework.

Methods

We evaluated 4 contemporary LLMs, GPT-4.1, GPT-5.2, Claude Sonnet 4.5, and Opus 4.6, against responses from 99 participants recruited through Prolific (Prolific Academic Ltd). Participants answered 10 questions for discharge summaries that varied in length and medical complexity. Questions included information-based items assessing factual comprehension and perception-based items assessing subjective understanding. Model-human alignment was measured using modal mean absolute error and expected absolute ordinal distance. We also conducted a chain-of-thought sensitivity analysis for cases with the greatest divergence between LLM-generated persona and matched human responses.

Results

Across LLMs, persona responses aligned more closely with human responses on information-based items than on perception-based items. Education was the most consistent stratification axis: every model aligned more closely with the high-education subgroup than with the low-education subgroup on information items, suggesting that LLMs capture broad factual-comprehension differences associated with educational background. Perception items, in which subjective understanding and uncertainty dominate human responses, were simulated less reliably. Alignment also depended on discharge summary characteristics, with short but clinically complex summaries producing the largest divergence. Gender-related differences were smaller and inconsistent in direction, reflecting item-level modal flips rather than systematic distributional bias. Chain-of-thought improved alignment on some factual items but did not recover the full range of less common human responses. Across models, LLMs underselected “I don’t know” and “Not provided” responses relative to humans, indicating a tendency to commit to determinate answers even when human participants expressed uncertainty.

Conclusions

Contemporary LLMs can approximate common response patterns within demographic subgroups, particularly for factual comprehension and education-based personas. However, they are less reliable for capturing heterogeneous and perception-based patient responses. Equity performance varied substantially across models, making model selection an important design consideration. Persona-based LLM simulation should be used as an assistive screening tool for patient materials and should complement, not replace, human evaluation with subgroup validation.

Subjects

MeSH terms (MEDLINE)

Large Language ModelsComprehensionPatient DischargePatient Discharge SummariesHumansFemaleMaleAdultMiddle Aged

Author keywords

AIartificial intelligencedigital healthfairness, bias, and ethics in AIhealth informaticsmachine learning

Authors and institutions

  • Luddy School of Informatics, Computing, and Engineering, Indiana University Bloomington, 700 N Woodlawn Ave, Bloomington, IN, United States, 1 812-223-3981.
  • School of Medicine, Yale University, New Haven, CT, United States.
  • College of Computing and Data Science, Nanyang Technological University, Singapore, Singapore.
  • Biostatistics, University of California, Davis, Davis, CA, United States.
  • College of Computing and Informatics, University of North Carolina at Charlotte, Charlotte, NC, United States.

Cite

APA

Ma, X., Zhu, R., Wang, Z., Xiong, J., Chen, Q., Camp, L. J., & Tang, H. (2026). Large Language Models’ Simulation of Patient Comprehension of Discharge Summaries: Multimodel Evaluation Study. Journal of Medical Internet Research, 28, e89003-e89003. https://doi.org/10.2196/89003

Vancouver

Ma X, Zhu R, Wang Z, Xiong J, Chen Q, Camp LJ, et al. Large Language Models’ Simulation of Patient Comprehension of Discharge Summaries: Multimodel Evaluation Study. Journal of Medical Internet Research. 2026;28:e89003-e89003. doi:10.2196/89003

BibTeX
@article{Ma2026,
  author = {Ma, Xinyao and Zhu, Rui and Wang, Zihao and Xiong, Jingwei and Chen, Qingyu and Camp, L Jean and Tang, Haixu},
  title = {Large Language Models’ Simulation of Patient Comprehension of Discharge Summaries: Multimodel Evaluation Study},
  journal = {Journal of Medical Internet Research},
  year = {2026},
  volume = {28},
  pages = {e89003--e89003},
  doi = {10.2196/89003}
}
Start for free →