EUVIMEDEUVIMED entdecken
Open Access

Topic-Aware Summarization of Lived Health Care Experiences: Large Language Model Evaluation Study

Maneesh Bilalpur iD, Megan E Hamm iD, Young Ji Lee iD, Natasha G Norman iD, Kathleen M Mctigue iD, Yanshan Wang iD

JMIR Medical Informatics · 2026

DOI: 10.2196/85960 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Medical Informatics, 2026. Verfasst von Maneesh Bilalpur und 5 weiteren Autor:innen. Beteiligte Einrichtung: Intelligent Systems Program, University of Pittsburgh, 4200 Fifth Avenue, Pittsburgh, PA, 15260, United States, 1 4123832712. Hauptthemen laut MEDLINE: Caregivers, Large Language Models, Narration, Health Behavior. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 59 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NCATS NIH HHS, NLM NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Existing work to understand adults’ health care experiences has focused on the analysis of patient feedback provided as written responses to after-visit surveys or social media discourse. Often, such written feedback has been studied using natural language processing techniques, such as topic detection and sentiment analysis, to provide coarse-grained insights. Storytelling is a powerful form of communication and may provide insights into factors contributing to gaps in health care outcomes and avenues for improvement. In addition, studying health care experiences using natural language processing techniques has been limited to patients. The experiences of stakeholders, such as caregivers and health care providers, remain underexplored.

Objective

We extract fine-grained insights from health care experiences through narratives collected from patients, caregivers, and health care providers using large language models (LLMs). Topic detection, together with hierarchical summarization of long-form stories from individuals, offers fine-grained insights. Furthermore, the study demonstrates that generated summaries can be evaluated using the LLM-as-a-judge framework and validates the outcomes through comparisons with 2 domain experts.

Methods

Fifty automatically transcribed stories of African American experiences were used to identify topics in their experiences using the latent Dirichlet allocation (LDA) technique. Stories about a given topic were summarized using an open-source LLM-based hierarchical summarization approach. Topic summaries were generated by summarizing across story summaries for each story that addressed a given topic. The generated topic summaries were rated for fabrication, accuracy, comprehensiveness, and usefulness by the GPT-4 model; its reliability was validated against the original story summaries by 2 domain experts.

Results

Whisper-based automatic transcription of audio narrations achieved a Levenshtein score of 6%. Twenty-six topics were identified using LDA and labeled using the LLM in the 50 African American stories. The GPT-4 ratings suggest that topic summaries were free from fabrication, highly accurate, comprehensive, and useful. The reliability of GPT ratings compared to expert assessments showed moderate-to-high agreement (Bennett S -score of 0.65 or higher). Our approach identified African American experience-relevant topics, such as health behaviors, interactions with medical team members, caregiving, and symptom management, among others. Such insights could help researchers learn from unstructured datasets in an efficient manner—leveraging the communicative power of storytelling.

Conclusions

The use of LDA and LLMs to identify and summarize the experiences of African American individuals suggests a variety of possible avenues for health research and possible clinical improvements to support patients and caregivers, thereby improving health outcomes.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

CaregiversLarge Language ModelsNarrationHealth BehaviorPatient SatisfactionHumansNatural Language ProcessingQualitative ResearchBlack or African American

Stichwörter der Autor:innen

health disparitieslarge language modelsnatural language processingtext summarizationtopic modelingunstructured qualitative data

Autor:innen und Institutionen

  • Intelligent Systems Program, University of Pittsburgh, 4200 Fifth Avenue, Pittsburgh, PA, 15260, United States, 1 4123832712.
  • Department of Medicine, University of Pittsburgh, Pittsburgh, PA, United States.
  • School of Nursing, University of Pittsburgh, Pittsburgh, PA, United States.
  • Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, PA, United States.
  • Department of Health Information Management, University of Pittsburgh, Pittsburgh, PA, United States.

ORCID-Profile: Maneesh Bilalpur, Megan E Hamm, Young Ji Lee, Natasha G Norman, Kathleen M Mctigue, Yanshan Wang

Förderung

  • NCATS NIH HHS
  • NLM NIH HHS

Zitieren

APA

Bilalpur, M., Hamm, M. E., Lee, Y. J., Norman, N. G., Mctigue, K. M., & Wang, Y. (2026). Topic-Aware Summarization of Lived Health Care Experiences: Large Language Model Evaluation Study. JMIR Medical Informatics. https://doi.org/10.2196/85960

Vancouver

Bilalpur M, Hamm ME, Lee YJ, Norman NG, Mctigue KM, Wang Y. Topic-Aware Summarization of Lived Health Care Experiences: Large Language Model Evaluation Study. JMIR Medical Informatics. 2026. doi:10.2196/85960

BibTeX
@article{Bilalpur2026,
  author = {Bilalpur, Maneesh and Hamm, Megan E and Lee, Young Ji and Norman, Natasha G and Mctigue, Kathleen M and Wang, Yanshan},
  title = {Topic-Aware Summarization of Lived Health Care Experiences: Large Language Model Evaluation Study},
  journal = {JMIR Medical Informatics},
  year = {2026},
  doi = {10.2196/85960}
}
Kostenlos starten →