EUVIMEDEUVIMED entdecken
Open Access

A dataset and benchmark for hospital course summarization with adapted large language models

Asad Aali iD, Dave Van Veen, Yamin Ishraq Arefeen, Jason Hom, Christian Bluethgen, Eduardo Pontes Reis, Sergios Gatidis, Namuun Clifford, Joseph Daws, Arash S Tehrani, Jangwon Kim, Akshay S Chaudhari

Journal of the American Medical Informatics Association · 01.03.2025 · 32 (3) · 470-479

DOI: 10.1093/jamia/ocae312 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 30.12.2024. Verfasst von Asad Aali und 11 weiteren Autor:innen. Beteiligte Einrichtung: Department of Radiology, Stanford University, Stanford, CA 94304, United States. Hauptthemen laut MEDLINE: Natural Language Processing, Electronic Health Records. Die Arbeit wurde bisher 37-mal zitiert (Crossref) und nennt 45 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIBIB NIH HHS, Stanford Center for Artificial Intelligence and Medicine, NHLBI NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

Brief hospital course (BHC) summaries are clinical documents that summarize a patient’s hospital stay. While large language models (LLMs) depict remarkable capabilities in automating real-world tasks, their capabilities for healthcare applications such as synthesizing BHCs from clinical notes have not been shown. We introduce a novel preprocessed dataset, the MIMIC-IV-BHC, encapsulating clinical note and BHC pairs to adapt LLMs for BHC synthesis. Furthermore, we introduce a benchmark of the summarization performance of 2 general-purpose LLMs and 3 healthcare-adapted LLMs. Materials and Methods Using clinical notes as input, we apply prompting-based (using in-context learning) and fine-tuning-based adaptation strategies to 3 open-source LLMs (Clinical-T5-Large, Llama2-13B, and FLAN-UL2) and 2 proprietary LLMs (Generative Pre-trained Transformer [GPT]-3.5 and GPT-4). We evaluate these LLMs across multiple context-length inputs using natural language similarity metrics. We further conduct a clinical study with 5 clinicians, comparing clinician-written and LLM-generated BHCs across 30 samples, focusing on their potential to enhance clinical decision-making through improved summary quality. We compare reader preferences for the original and LLM-generated summary using Wilcoxon signed-rank tests. We further request optional qualitative feedback from clinicians to gain deeper insights into their preferences, and we present the frequency of common themes arising from these comments.

Results

The Llama2-13B fine-tuned LLM outperforms other domain-adapted models given quantitative evaluation metrics of Bilingual Evaluation Understudy (BLEU) and Bidirectional Encoder Representations from Transformers (BERT)-Score. GPT-4 with in-context learning shows more robustness to increasing context lengths of clinical note inputs than fine-tuned Llama2-13B. Despite comparable quantitative metrics, the reader study depicts a significant preference for summaries generated by GPT-4 with in-context learning compared to both Llama2-13B fine-tuned summaries and the original summaries (P<.001), highlighting the need for qualitative clinical evaluation. Discussion and Conclusion We release a foundational clinically relevant dataset, the MIMIC-IV-BHC, and present an open-source benchmark of LLM performance in BHC synthesis from clinical notes. We observe high-quality summarization performance for both in-context proprietary and fine-tuned open-source LLMs using both quantitative metrics and a qualitative clinical reader study. Our research effectively integrates elements from the data assimilation pipeline: our methods use (1) clinical data sources to integrate, (2) data translation, and (3) knowledge creation, while our evaluation strategy paves the way for (4) deployment.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language ProcessingElectronic Health RecordsHumansBenchmarkingDatasets as TopicLarge Language Models

Stichwörter der Autor:innen

electronic health recordsinformation storage and retrievalmachine learningnatural language processing

Autor:innen und Institutionen

  • Department of Radiology, Stanford University, Stanford, CA 94304, United States.
  • Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX 78712, United States.
  • Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, CA 94304, United States.
  • Department of Electrical Engineering, Stanford University, Stanford, CA 94304, United States.
  • Department of Medicine, Stanford University, Stanford, CA 94304, United States.
  • University Hospital Zurich, Zurich 8091, Switzerland.

ORCID-Profile: Asad Aali

Förderung

  • NIBIB NIH HHS
  • Stanford Center for Artificial Intelligence and Medicine
  • NHLBI NIH HHS
  • NINR NIH HHS
  • Stanford Center for Digital Health
  • NIAMS NIH HHS
  • ProMedica Foundation
  • Accelerate Foundation Models Academic Research

Zitieren

APA

Aali, A., Van Veen, D., Arefeen, Y. I., Hom, J., Bluethgen, C., Reis, E. P., Gatidis, S., Clifford, N., Daws, J., Tehrani, A. S., Kim, J., & Chaudhari, A. S. (2025). A dataset and benchmark for hospital course summarization with adapted large language models. Journal of the American Medical Informatics Association, 32(3), 470-479. https://doi.org/10.1093/jamia/ocae312

Vancouver

Aali A, Van Veen D, Arefeen YI, Hom J, Bluethgen C, Reis EP, et al. A dataset and benchmark for hospital course summarization with adapted large language models. Journal of the American Medical Informatics Association. 2025;32(3):470-479. doi:10.1093/jamia/ocae312

BibTeX
@article{Aali2025,
  author = {Aali, Asad and Van Veen, Dave and Arefeen, Yamin Ishraq and Hom, Jason and Bluethgen, Christian and Reis, Eduardo Pontes and Gatidis, Sergios and Clifford, Namuun and Daws, Joseph and Tehrani, Arash S and Kim, Jangwon and Chaudhari, Akshay S},
  title = {A dataset and benchmark for hospital course summarization with adapted large language models},
  journal = {Journal of the American Medical Informatics Association},
  year = {2025},
  volume = {32},
  number = {3},
  pages = {470--479},
  doi = {10.1093/jamia/ocae312}
}
Kostenlos starten →