Topic-Aware Summarization of Lived Health Care Experiences: Large Language Model Evaluation Study
JMIR Medical Informatics · 2026
DOI: 10.2196/85960 ↗Kurz gefasst
Fachartikel in der Zeitschrift JMIR Medical Informatics, 2026. Verfasst von Maneesh Bilalpur und 5 weiteren Autor:innen. Beteiligte Einrichtung: Intelligent Systems Program, University of Pittsburgh, 4200 Fifth Avenue, Pittsburgh, PA, 15260, United States, 1 4123832712. Hauptthemen laut MEDLINE: Caregivers, Large Language Models, Narration, Health Behavior. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 59 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NCATS NIH HHS, NLM NIH HHS.
Abstract (Originalsprache)
Background
Existing work to understand adults’ health care experiences has focused on the analysis of patient feedback provided as written responses to after-visit surveys or social media discourse. Often, such written feedback has been studied using natural language processing techniques, such as topic detection and sentiment analysis, to provide coarse-grained insights. Storytelling is a powerful form of communication and may provide insights into factors contributing to gaps in health care outcomes and avenues for improvement. In addition, studying health care experiences using natural language processing techniques has been limited to patients. The experiences of stakeholders, such as caregivers and health care providers, remain underexplored.
Objective
We extract fine-grained insights from health care experiences through narratives collected from patients, caregivers, and health care providers using large language models (LLMs). Topic detection, together with hierarchical summarization of long-form stories from individuals, offers fine-grained insights. Furthermore, the study demonstrates that generated summaries can be evaluated using the LLM-as-a-judge framework and validates the outcomes through comparisons with 2 domain experts.
Methods
Fifty automatically transcribed stories of African American experiences were used to identify topics in their experiences using the latent Dirichlet allocation (LDA) technique. Stories about a given topic were summarized using an open-source LLM-based hierarchical summarization approach. Topic summaries were generated by summarizing across story summaries for each story that addressed a given topic. The generated topic summaries were rated for fabrication, accuracy, comprehensiveness, and usefulness by the GPT-4 model; its reliability was validated against the original story summaries by 2 domain experts.
Results
Whisper-based automatic transcription of audio narrations achieved a Levenshtein score of 6%. Twenty-six topics were identified using LDA and labeled using the LLM in the 50 African American stories. The GPT-4 ratings suggest that topic summaries were free from fabrication, highly accurate, comprehensive, and useful. The reliability of GPT ratings compared to expert assessments showed moderate-to-high agreement (Bennett S -score of 0.65 or higher). Our approach identified African American experience-relevant topics, such as health behaviors, interactions with medical team members, caregiving, and symptom management, among others. Such insights could help researchers learn from unstructured datasets in an efficient manner—leveraging the communicative power of storytelling.
Conclusions
The use of LDA and LLMs to identify and summarize the experiences of African American individuals suggests a variety of possible avenues for health research and possible clinical improvements to support patients and caregivers, thereby improving health outcomes.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Intelligent Systems Program, University of Pittsburgh, 4200 Fifth Avenue, Pittsburgh, PA, 15260, United States, 1 4123832712.
- Department of Medicine, University of Pittsburgh, Pittsburgh, PA, United States.
- School of Nursing, University of Pittsburgh, Pittsburgh, PA, United States.
- Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, PA, United States.
- Department of Health Information Management, University of Pittsburgh, Pittsburgh, PA, United States.
ORCID-Profile: Maneesh Bilalpur, Megan E Hamm, Young Ji Lee, Natasha G Norman, Kathleen M Mctigue, Yanshan Wang
Förderung
- NCATS NIH HHS
- NLM NIH HHS
Zitieren
APA
Bilalpur, M., Hamm, M. E., Lee, Y. J., Norman, N. G., Mctigue, K. M., & Wang, Y. (2026). Topic-Aware Summarization of Lived Health Care Experiences: Large Language Model Evaluation Study. JMIR Medical Informatics. https://doi.org/10.2196/85960
Vancouver
Bilalpur M, Hamm ME, Lee YJ, Norman NG, Mctigue KM, Wang Y. Topic-Aware Summarization of Lived Health Care Experiences: Large Language Model Evaluation Study. JMIR Medical Informatics. 2026. doi:10.2196/85960
BibTeX
@article{Bilalpur2026,
author = {Bilalpur, Maneesh and Hamm, Megan E and Lee, Young Ji and Norman, Natasha G and Mctigue, Kathleen M and Wang, Yanshan},
title = {Topic-Aware Summarization of Lived Health Care Experiences: Large Language Model Evaluation Study},
journal = {JMIR Medical Informatics},
year = {2026},
doi = {10.2196/85960}
}
EUVIMED