Leveraging a Large Language Model to Generate Quality Improvement Feedback for Clinical Notes
Applied Clinical Informatics · 01.03.2026 · 17 (02) · 252-258
DOI: 10.1055/a-2851-0739 ↗Kurz gefasst
Fachartikel in der Zeitschrift Applied Clinical Informatics, erschienen am 01.03.2026. Verfasst von Christopher J Kim und 14 weiteren Autor:innen. Beteiligte Einrichtung: Division of Hospital Medicine, Department of Medicine, NYU Langone Health, New York, New York, United States. Hauptthemen laut MEDLINE: Large Language Models, Quality Improvement, Feedback, Documentation. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Poor documentation quality can significantly affect health care operations, but the feedback process for clinicians to improve clinical notes is time-consuming and often insufficient. Large language models (LLMs) such as Generative Pre-trained Transformer 4 (GPT-4) have the potential to streamline this process.
Objectives
This study aimed to determine whether an LLM can generate feedback to improve the medical contingency and discharge planning (MCDP) component of clinical documentation that is non-inferior to feedback by physicians.
Methods
A cross-sectional study of GPT-4 feedback and physician feedback on inpatient progress notes was conducted. A random sample of 64 inpatient progress notes identified by the validated artificial intelligence (AI) Audit Tool as having a low likelihood of containing MCDP was included from adult general medicine patients hospitalized at New York University Langone Health (NYULH) in December 2023. Both the GPT-4 model and attending physicians generated feedback on these inpatient progress notes. A/B testing was then conducted on the measures of understandability, usefulness, acceptability, and impartiality. Evaluations employed 5-point Likert scales that were converted to 10-point bidirectional interval scales for interpretability, ranging from −10 (human suggestions significantly better) to +10 (GPT-4 suggestions significantly better), with a non-inferiority threshold set to −1 for the primary endpoint.
Results
Sixty-four inpatient progress notes were included, representing 55% female patients with a median age of 73 years. GPT-4 feedback was non-inferior to physician feedback in all measures: Understandability (mean, 1.27, 95% CI: 0.73–1.8, p < 0.001), usefulness (mean, 2.09, 95% CI: 1.27–2.91, p < 0.001), acceptability (mean, 2.07, 95% CI: 1.33–2.81, p < 0.001), and impartiality (mean, −0.20, 95% CI: −0.52 to 0.12, p < 0.001).
Conclusion
This study shows that an LLM can be leveraged to generate note quality feedback that is non-inferior to expert clinician feedback.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Division of Hospital Medicine, Department of Medicine, NYU Langone Health, New York, New York, United States.
- Department of Medicine, NYU Grossman Long Island School of Medicine, Mineola, New York, United States.
- Department of Foundations of Medicine, NYU Grossman Long Island School of Medicine, Mineola, New York, United States.
- Department of Medicine, NYU Grossman School of Medicine, New York, New York, United States.
- Department of Health Informatics, Medical Center Information Technology, NYU Langone Health, New York, New York, United States.
- Division of Gastroenterology and Hepatology, Department of Medicine, NYU Langone Health, New York, New York, United States.
Zitieren
APA
Kim, C. J., Gelfinbein, J., Gencerliler, N., Jahan, N., Udaikumar, J., Heery, L. M., Goodman, A., Ng, S., Attard, J., Asha, S., Burk-Rafel, J., Guzman, B. V., Hochman, K. A., Testa, P. A., & Feldman, J. (2026). Leveraging a Large Language Model to Generate Quality Improvement Feedback for Clinical Notes. Applied Clinical Informatics, 17(02), 252-258. https://doi.org/10.1055/a-2851-0739
Vancouver
Kim CJ, Gelfinbein J, Gencerliler N, Jahan N, Udaikumar J, Heery LM, et al. Leveraging a Large Language Model to Generate Quality Improvement Feedback for Clinical Notes. Applied Clinical Informatics. 2026;17(02):252-258. doi:10.1055/a-2851-0739
BibTeX
@article{Kim2026,
author = {Kim, Christopher J and Gelfinbein, Joseph and Gencerliler, Nihan and Jahan, Nusrat and Udaikumar, Jahnavi and Heery, Lauren M and Goodman, Adam and Ng, Sarah and Attard, Joel and Asha, Sharmin and Burk-Rafel, Jesse and Guzman, Benedict V and Hochman, Katherine A and Testa, Paul A and Feldman, Jonah},
title = {Leveraging a Large Language Model to Generate Quality Improvement Feedback for Clinical Notes},
journal = {Applied Clinical Informatics},
year = {2026},
volume = {17},
number = {02},
pages = {252--258},
doi = {10.1055/a-2851-0739}
}
EUVIMED