Evaluating Hospital Course Summarization by an Electronic Health Record–Based Large Language Model
JAMA Network Open · 13.08.2025 · 8 (8) · e2526339
DOI: 10.1001/jamanetworkopen.2025.26339 ↗Kurz gefasst
Fachartikel in der Zeitschrift JAMA Network Open, online erschienen am 13.08.2025. Verfasst von William R Small und 30 weiteren Autor:innen. Beteiligte Einrichtung: Department of Health Informatics, New York University Langone Medical Center Information Technology. Hauptthemen laut MEDLINE: Electronic Health Records, Language. Die Arbeit wurde bisher 45-mal zitiert (Crossref) und nennt 57 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Importance Hospital course (HC) summarization represents an increasingly onerous discharge summary component for physicians. Literature supports large language models (LLMs) for HC summarization, but whether physicians can effectively partner with electronic health record–embedded LLMs to draft HCs is unknown.
Objectives
To compare the editing effort required by time-constrained resident physicians to improve LLM- vs physician-generated HCs toward a novel 4Cs (complete, concise, cohesive, and confabulation-free) HC. Design, Setting, and Participants Quality improvement study using a convenience sample of 10 internal medicine resident editors, 8 hospitalist evaluators, and randomly selected general medicine admissions in December 2023 lasting 4 to 8 days at New York University Langone Health. Exposures Residents and hospitalists reviewed randomly assigned patient medical records for 10 minutes. Residents blinded to author type who edited each HC pair (physician and LLM) for quality in 3 minutes, followed by comparative ratings by attending hospitalists. Main Outcomes and Measures Editing effort was quantified by analyzing the edits that occurred on the HC pairs after controlling for length (percentage edited) and the degree to which the original HCs’ meaning was altered (semantic change). Hospitalists compared edited HC pairs with A/B testing on the 4Cs (5-point Likert scales converted to 10-point bidirectional scales).
Results
Among 100 admissions, compared with physician HCs, residents edited a smaller percentage of LLM HCs (LLM mean [SD], 31.5% [16.6%] vs physicians, 44.8% [20.0%]; P < .001). Additionally, LLM HCs required less semantic change (LLM mean [SD], 2.4% [1.6%] vs physicians, 4.9% [3.5%]; P < .001). Attending physicians deemed LLM HCs to be more complete (mean [SD] difference LLM vs physicians on 10-point bidirectional scale, 3.00 [5.28]; P < .001), similarly concise (mean [SD], −1.02 [6.08]; P = .20), and cohesive (mean [SD], 0.70 [6.14]; P = .60), but with more confabulations (mean [SD], −0.98 [3.53]; P = .002). The composite scores were similar (mean [SD] difference LLM vs physician on 40-point bidirectional scale, 1.70 [14.24]; P = .46). Conclusions and Relevance Electronic health record–embedded LLM HCs required less editing than physician-generated HCs to approach a quality standard, resulting in HCs that were comparably or more complete, concise, and cohesive, but contained more confabulations. Despite the potential influence of artificial time constraints, this study supports the feasibility of a physician-LLM partnership for writing HCs and provides a basis for monitoring LLM HCs in clinical practice.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Department of Health Informatics, New York University Langone Medical Center Information Technology.
- Department of Medicine, New York University Grossman School of Medicine.
- Department of Population Health, New York University Grossman School of Medicine.
- Epic Systems Corporation, Madison, Wisconsin.
- Department of Medicine, New York University Grossman Long Island School of Medicine, Mineola.
- Department of Emergency Medicine, New York University Grossman School of Medicine.
Zitieren
APA
Small, W. R., Austrian, J., O'Donnell, L., Burk-Rafel, J., Hochman, K. A., Goodman, A., Zaretsky, J., Martin, J., Johnson, S., Major, V. J., Jones, S., Henke, C., Verplanke, B., Osso, J., Larson, I., Saxena, A., Mednick, A., Simonis, C., Han, J., …, & Feldman, J. (2025). Evaluating Hospital Course Summarization by an Electronic Health Record–Based Large Language Model. JAMA Network Open, 8(8), e2526339. https://doi.org/10.1001/jamanetworkopen.2025.26339
Vancouver
Small WR, Austrian J, O'Donnell L, Burk-Rafel J, Hochman KA, Goodman A, et al. Evaluating Hospital Course Summarization by an Electronic Health Record–Based Large Language Model. JAMA Network Open. 2025;8(8):e2526339. doi:10.1001/jamanetworkopen.2025.26339
BibTeX
@article{Small2025,
author = {Small, William R and Austrian, Jonathan and O'Donnell, Luke and Burk-Rafel, Jesse and Hochman, Katherine A and Goodman, Adam and Zaretsky, Jonah and Martin, Jacob and Johnson, Stephen and Major, Vincent J and Jones, Simon and Henke, Christian and Verplanke, Benjamin and Osso, Jwan and Larson, Ian and Saxena, Archana and Mednick, Aron and Simonis, Choumika and Han, Joseph and Kesari, Ravi and Wu, Xinyuan and Heery, Lauren and Desel, Tenzin and Baskharoun, Samuel and Figman, Noah and Farooq, Umar and Shah, Kunal and Jahan, Nusrat and Kim, Jeong Min and Testa, Paul and Feldman, Jonah},
title = {Evaluating Hospital Course Summarization by an Electronic Health Record–Based Large Language Model},
journal = {JAMA Network Open},
year = {2025},
volume = {8},
number = {8},
pages = {e2526339},
doi = {10.1001/jamanetworkopen.2025.26339}
}
EUVIMED