Testing and evaluation of generative large language models in electronic health record applications: a systematic review
Journal of the American Medical Informatics Association · 01.03.2026 · 33 (3) · 743-753
DOI: 10.1093/jamia/ocaf233 ↗Kurz gefasst
Systematische Übersicht in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 13.01.2026. Verfasst von Xinsong Du und 13 weiteren Autor:innen. Beteiligte Einrichtung: Division of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Boston, MA 02115, United States. Hauptthemen laut MEDLINE: Large Language Models, Electronic Health Records. Die Arbeit wurde bisher 32-mal zitiert (Crossref) und nennt 57 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIA NIH HHS, NLM NIH HHS, National Institute of Health-National Institute of Aging.
Abstract (Originalsprache)
Background
The use of generative large language models (LLMs) with electronic health record (EHR) data is rapidly expanding to support clinical and research tasks. This systematic review characterizes the clinical fields and use cases that have been studied and evaluated to date.
Methods
We followed the Preferred Reporting Items for Systematic Review and Meta-Analyses guidelines to conduct a systematic review of articles from PubMed and Web of Science published between January 1, 2023, and November 9, 2024. Studies were included if they used generative LLMs to analyze real-world EHR data and reported quantitative performance evaluations. Through data extraction, we identified clinical specialties and tasks for each included article, and summarized evaluation methods.
Results
Of the 18 735 articles retrieved, 196 met our criteria. Most studies focused on radiology (26.0%), oncology (10.7%), and emergency medicine (6.6%). Regarding clinical tasks, clinical decision support made up the largest proportion of studies (62.2%), while summarizations and patient communications made up the smallest, at 5.6% and 5.1%, respectively. In addition, GPT-4 and GPT-3.5 were the most commonly used generative LLMs, appearing in 60.2% and 57.7% of studies, respectively. Across these studies, we identified 22 unique non-NLP metrics and 35 unique NLP metrics. While NLP metrics offer greater scalability, none demonstrated a strong correlation with gold-standard human evaluations.
Conclusion
Our findings highlight the need to evaluate generative LLMs on EHR data across a broader range of clinical specialties and tasks, as well as the urgent need for standardized, scalable, and clinically meaningful evaluation frameworks.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Division of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Boston, MA 02115, United States.
- Department of Medicine, Harvard Medical School, Boston, MA 02115, United States.
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA 02115, United States.
- Department of Computer Science, Brandeis University, Waltham, MA 02453, United States.
- Division of Nephrology, Department of Internal Medicine, Taichung Veterans General Hospital, Taichung 407219, Taiwan.
- Department of Post-Baccalaureate Medicine, College of Medicine, National Chung Hsing University, Taichung 402202, Taiwan.
ORCID-Profile: Xinsong Du, Yiming Li, David W Bates
Förderung
- NIA NIH HHS
- NLM NIH HHS
- National Institute of Health-National Institute of Aging
- National Institute of Health-National Library of Medicine
Zitieren
APA
Du, X., Zhou, Z., Wang, Y., Chuang, Y. W., Li, Y., Yang, R., Zhang, W., Wang, X., Chen, X., Guan, H., Lian, J., Hong, P., Bates, D. W., & Zhou, L. (2026). Testing and evaluation of generative large language models in electronic health record applications: a systematic review. Journal of the American Medical Informatics Association, 33(3), 743-753. https://doi.org/10.1093/jamia/ocaf233
Vancouver
Du X, Zhou Z, Wang Y, Chuang YW, Li Y, Yang R, et al. Testing and evaluation of generative large language models in electronic health record applications: a systematic review. Journal of the American Medical Informatics Association. 2026;33(3):743-753. doi:10.1093/jamia/ocaf233
BibTeX
@article{Du2026,
author = {Du, Xinsong and Zhou, Zhengyang and Wang, Yifei and Chuang, Ya-Wen and Li, Yiming and Yang, Richard and Zhang, Wenyu and Wang, Xinyi and Chen, Xinyu and Guan, Hao and Lian, John and Hong, Pengyu and Bates, David W and Zhou, Li},
title = {Testing and evaluation of generative large language models in electronic health record applications: a systematic review},
journal = {Journal of the American Medical Informatics Association},
year = {2026},
volume = {33},
number = {3},
pages = {743--753},
doi = {10.1093/jamia/ocaf233}
}
EUVIMED