Development and validation of the provider documentation summarization quality instrument for large language models
Journal of the American Medical Informatics Association · 01.06.2025 · 32 (6) · 1050-1060
DOI: 10.1093/jamia/ocaf068 ↗Kurz gefasst
Validierungsstudie in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 05.05.2025. Verfasst von Emma Croxford und 18 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biostatistics and Medical Informatics, University of Wisconsin, Madison, WI 53792, United States. Hauptthemen laut MEDLINE: Electronic Health Records, Documentation. Die Arbeit wurde bisher 42-mal zitiert (Crossref) und nennt 55 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NLM NIH HHS, NCI NIH HHS, National Institute of Health.
Abstract (Originalsprache)
Objectives
As large language models (LLMs) are integrated into electronic health record (EHR) workflows, validated instruments are essential to evaluate their performance before implementation and as models and documentation practices evolve. Existing instruments for provider documentation quality are often unsuitable for the complexities of LLM-generated text and lack validation on real-world data. The Provider Documentation Summarization Quality Instrument (PDSQI-9) was developed to evaluate LLM-generated clinical summaries. This study aimed to validate the PDSQI-9 across key aspects of construct validity. Materials and Methods Multi-document summaries were generated from real-world EHR data across multiple specialties using several LLMs (GPT-4o, Mixtral 8x7b, and Llama 3-8b). Validation included Pearson correlation analyses for substantive validity, factor analysis and Cronbach’s α for structural validity, inter-rater reliability (ICC and Krippendorff’s α) for generalizability, a semi-Delphi process for content validity, and comparisons of high- versus low-quality summaries for discriminant validity. Raters underwent standardized training to ensure consistent application of the instrument.
Results
Seven physician raters evaluated 779 summaries and answered 8329 questions, achieving over 80% power for inter-rater reliability. The PDSQI-9 demonstrated strong internal consistency (Cronbach’s α = 0.879; 95% CI, 0.867-0.891) and high inter-rater reliability (ICC = 0.867; 95% CI, 0.867-0.868), supporting structural validity and generalizability. Factor analysis identified a 4-factor model explaining 58% of the variance, representing organization, clarity, accuracy, and utility. Substantive validity was supported by correlations between note length and scores for Succinct (ρ = −0.200, P = .029) and Organized (ρ = −0.190, P = .037). The semi-Delphi process ensured clinically relevant attributes, and discriminant validity distinguished high- from low-quality summaries (P<.001).
Discussion
The PDSQI-9 showed high inter-rater reliability, internal consistency, and a meaningful factor structure that reliably captured key dimensions of documentation quality. It distinguished between high- and low-quality summaries, supporting its practical utility for health systems needing an evaluation instrument for LLMs.
Conclusions
The PDSQI-9 demonstrates robust construct validity, supporting its use in clinical practice to evaluate LLM-generated summaries and facilitate safer, more effective integration of LLMs into healthcare workflows.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Biostatistics and Medical Informatics, University of Wisconsin, Madison, WI 53792, United States.
- Department of Biomedical Informatics, University of Colorado-Anschutz Medical, Aurora, CO 80045, United States.
- Epic Systems, Verona, WI 53593, United States.
- UW Health, Madison, WI 53726, United States.
- Department of Pediatrics, University of Wisconsin, Madison, WI 53792, United States.
- BerbeeWalsh Department of Emergency Medicine, University of Wisconsin, Madison, WI 53792, United States.
ORCID-Profile: Yanjun Gao, Marie Pisani, Frank Liao, Brian Patterson, Majid Afshar
Förderung
- NLM NIH HHS
- NCI NIH HHS
- National Institute of Health
- National Library of Medicine
Zitieren
APA
Croxford, E., Gao, Y., Pellegrino, N., Wong, K., Wills, G., First, E., Schnier, M., Burton, K., Ebby, C., Gorski, J., Kalscheur, M., Khalil, S., Pisani, M., Rubeor, T., Stetson, P., Liao, F., Goswami, C., Patterson, B., & Afshar, M. (2025). Development and validation of the provider documentation summarization quality instrument for large language models. Journal of the American Medical Informatics Association, 32(6), 1050-1060. https://doi.org/10.1093/jamia/ocaf068
Vancouver
Croxford E, Gao Y, Pellegrino N, Wong K, Wills G, First E, et al. Development and validation of the provider documentation summarization quality instrument for large language models. Journal of the American Medical Informatics Association. 2025;32(6):1050-1060. doi:10.1093/jamia/ocaf068
BibTeX
@article{Croxford2025,
author = {Croxford, Emma and Gao, Yanjun and Pellegrino, Nicholas and Wong, Karen and Wills, Graham and First, Elliot and Schnier, Miranda and Burton, Kyle and Ebby, Cris and Gorski, Jillian and Kalscheur, Matthew and Khalil, Samy and Pisani, Marie and Rubeor, Tyler and Stetson, Peter and Liao, Frank and Goswami, Cherodeep and Patterson, Brian and Afshar, Majid},
title = {Development and validation of the provider documentation summarization quality instrument for large language models},
journal = {Journal of the American Medical Informatics Association},
year = {2025},
volume = {32},
number = {6},
pages = {1050--1060},
doi = {10.1093/jamia/ocaf068}
}
EUVIMED