EUVIMEDEUVIMED entdecken
Open Access

Evaluating Hospital Course Summarization by an Electronic Health Record–Based Large Language Model

William R Small, Jonathan Austrian, Luke O'Donnell, Jesse Burk-Rafel, Katherine A Hochman, Adam Goodman, Jonah Zaretsky, Jacob Martin, Stephen Johnson, Vincent J Major, Simon Jones, Christian Henke und 19 weitere

JAMA Network Open · 13.08.2025 · 8 (8) · e2526339

DOI: 10.1001/jamanetworkopen.2025.26339 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JAMA Network Open, online erschienen am 13.08.2025. Verfasst von William R Small und 30 weiteren Autor:innen. Beteiligte Einrichtung: Department of Health Informatics, New York University Langone Medical Center Information Technology. Hauptthemen laut MEDLINE: Electronic Health Records, Language. Die Arbeit wurde bisher 45-mal zitiert (Crossref) und nennt 57 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Importance Hospital course (HC) summarization represents an increasingly onerous discharge summary component for physicians. Literature supports large language models (LLMs) for HC summarization, but whether physicians can effectively partner with electronic health record–embedded LLMs to draft HCs is unknown.

Objectives

To compare the editing effort required by time-constrained resident physicians to improve LLM- vs physician-generated HCs toward a novel 4Cs (complete, concise, cohesive, and confabulation-free) HC. Design, Setting, and Participants Quality improvement study using a convenience sample of 10 internal medicine resident editors, 8 hospitalist evaluators, and randomly selected general medicine admissions in December 2023 lasting 4 to 8 days at New York University Langone Health. Exposures Residents and hospitalists reviewed randomly assigned patient medical records for 10 minutes. Residents blinded to author type who edited each HC pair (physician and LLM) for quality in 3 minutes, followed by comparative ratings by attending hospitalists. Main Outcomes and Measures Editing effort was quantified by analyzing the edits that occurred on the HC pairs after controlling for length (percentage edited) and the degree to which the original HCs’ meaning was altered (semantic change). Hospitalists compared edited HC pairs with A/B testing on the 4Cs (5-point Likert scales converted to 10-point bidirectional scales).

Results

Among 100 admissions, compared with physician HCs, residents edited a smaller percentage of LLM HCs (LLM mean [SD], 31.5% [16.6%] vs physicians, 44.8% [20.0%]; P < .001). Additionally, LLM HCs required less semantic change (LLM mean [SD], 2.4% [1.6%] vs physicians, 4.9% [3.5%]; P < .001). Attending physicians deemed LLM HCs to be more complete (mean [SD] difference LLM vs physicians on 10-point bidirectional scale, 3.00 [5.28]; P < .001), similarly concise (mean [SD], −1.02 [6.08]; P = .20), and cohesive (mean [SD], 0.70 [6.14]; P = .60), but with more confabulations (mean [SD], −0.98 [3.53]; P = .002). The composite scores were similar (mean [SD] difference LLM vs physician on 40-point bidirectional scale, 1.70 [14.24]; P = .46). Conclusions and Relevance Electronic health record–embedded LLM HCs required less editing than physician-generated HCs to approach a quality standard, resulting in HCs that were comparably or more complete, concise, and cohesive, but contained more confabulations. Despite the potential influence of artificial time constraints, this study supports the feasibility of a physician-LLM partnership for writing HCs and provides a basis for monitoring LLM HCs in clinical practice.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Electronic Health RecordsLanguageHumansInternship and ResidencyQuality ImprovementMaleFemaleHospitalistsLarge Language Models

Autor:innen und Institutionen

  • Department of Health Informatics, New York University Langone Medical Center Information Technology.
  • Department of Medicine, New York University Grossman School of Medicine.
  • Department of Population Health, New York University Grossman School of Medicine.
  • Epic Systems Corporation, Madison, Wisconsin.
  • Department of Medicine, New York University Grossman Long Island School of Medicine, Mineola.
  • Department of Emergency Medicine, New York University Grossman School of Medicine.

Zitieren

APA

Small, W. R., Austrian, J., O'Donnell, L., Burk-Rafel, J., Hochman, K. A., Goodman, A., Zaretsky, J., Martin, J., Johnson, S., Major, V. J., Jones, S., Henke, C., Verplanke, B., Osso, J., Larson, I., Saxena, A., Mednick, A., Simonis, C., Han, J., …, & Feldman, J. (2025). Evaluating Hospital Course Summarization by an Electronic Health Record–Based Large Language Model. JAMA Network Open, 8(8), e2526339. https://doi.org/10.1001/jamanetworkopen.2025.26339

Vancouver

Small WR, Austrian J, O'Donnell L, Burk-Rafel J, Hochman KA, Goodman A, et al. Evaluating Hospital Course Summarization by an Electronic Health Record–Based Large Language Model. JAMA Network Open. 2025;8(8):e2526339. doi:10.1001/jamanetworkopen.2025.26339

BibTeX
@article{Small2025,
  author = {Small, William R and Austrian, Jonathan and O'Donnell, Luke and Burk-Rafel, Jesse and Hochman, Katherine A and Goodman, Adam and Zaretsky, Jonah and Martin, Jacob and Johnson, Stephen and Major, Vincent J and Jones, Simon and Henke, Christian and Verplanke, Benjamin and Osso, Jwan and Larson, Ian and Saxena, Archana and Mednick, Aron and Simonis, Choumika and Han, Joseph and Kesari, Ravi and Wu, Xinyuan and Heery, Lauren and Desel, Tenzin and Baskharoun, Samuel and Figman, Noah and Farooq, Umar and Shah, Kunal and Jahan, Nusrat and Kim, Jeong Min and Testa, Paul and Feldman, Jonah},
  title = {Evaluating Hospital Course Summarization by an Electronic Health Record–Based Large Language Model},
  journal = {JAMA Network Open},
  year = {2025},
  volume = {8},
  number = {8},
  pages = {e2526339},
  doi = {10.1001/jamanetworkopen.2025.26339}
}
Kostenlos starten →