EUVIMEDEUVIMED entdecken
Systematische ÜbersichtOpen Access

Testing and evaluation of generative large language models in electronic health record applications: a systematic review

Xinsong Du iD, Zhengyang Zhou, Yifei Wang, Ya-Wen Chuang, Yiming Li iD, Richard Yang, Wenyu Zhang, Xinyi Wang, Xinyu Chen, Hao Guan, John Lian, Pengyu Hong und 2 weitere

Journal of the American Medical Informatics Association · 01.03.2026 · 33 (3) · 743-753

DOI: 10.1093/jamia/ocaf233 ↗

Kurz gefasst

Systematische Übersicht in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 13.01.2026. Verfasst von Xinsong Du und 13 weiteren Autor:innen. Beteiligte Einrichtung: Division of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Boston, MA 02115, United States. Hauptthemen laut MEDLINE: Large Language Models, Electronic Health Records. Die Arbeit wurde bisher 32-mal zitiert (Crossref) und nennt 57 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIA NIH HHS, NLM NIH HHS, National Institute of Health-National Institute of Aging.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

The use of generative large language models (LLMs) with electronic health record (EHR) data is rapidly expanding to support clinical and research tasks. This systematic review characterizes the clinical fields and use cases that have been studied and evaluated to date.

Methods

We followed the Preferred Reporting Items for Systematic Review and Meta-Analyses guidelines to conduct a systematic review of articles from PubMed and Web of Science published between January 1, 2023, and November 9, 2024. Studies were included if they used generative LLMs to analyze real-world EHR data and reported quantitative performance evaluations. Through data extraction, we identified clinical specialties and tasks for each included article, and summarized evaluation methods.

Results

Of the 18 735 articles retrieved, 196 met our criteria. Most studies focused on radiology (26.0%), oncology (10.7%), and emergency medicine (6.6%). Regarding clinical tasks, clinical decision support made up the largest proportion of studies (62.2%), while summarizations and patient communications made up the smallest, at 5.6% and 5.1%, respectively. In addition, GPT-4 and GPT-3.5 were the most commonly used generative LLMs, appearing in 60.2% and 57.7% of studies, respectively. Across these studies, we identified 22 unique non-NLP metrics and 35 unique NLP metrics. While NLP metrics offer greater scalability, none demonstrated a strong correlation with gold-standard human evaluations.

Conclusion

Our findings highlight the need to evaluate generative LLMs on EHR data across a broader range of clinical specialties and tasks, as well as the urgent need for standardized, scalable, and clinically meaningful evaluation frameworks.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsElectronic Health RecordsHumansGenerative Artificial Intelligence

Stichwörter der Autor:innen

electronic health recordsevaluationlarge language modelsnatural language processingreview

Autor:innen und Institutionen

  • Division of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Boston, MA 02115, United States.
  • Department of Medicine, Harvard Medical School, Boston, MA 02115, United States.
  • Department of Biomedical Informatics, Harvard Medical School, Boston, MA 02115, United States.
  • Department of Computer Science, Brandeis University, Waltham, MA 02453, United States.
  • Division of Nephrology, Department of Internal Medicine, Taichung Veterans General Hospital, Taichung 407219, Taiwan.
  • Department of Post-Baccalaureate Medicine, College of Medicine, National Chung Hsing University, Taichung 402202, Taiwan.

ORCID-Profile: Xinsong Du, Yiming Li, David W Bates

Förderung

  • NIA NIH HHS
  • NLM NIH HHS
  • National Institute of Health-National Institute of Aging
  • National Institute of Health-National Library of Medicine

Zitieren

APA

Du, X., Zhou, Z., Wang, Y., Chuang, Y. W., Li, Y., Yang, R., Zhang, W., Wang, X., Chen, X., Guan, H., Lian, J., Hong, P., Bates, D. W., & Zhou, L. (2026). Testing and evaluation of generative large language models in electronic health record applications: a systematic review. Journal of the American Medical Informatics Association, 33(3), 743-753. https://doi.org/10.1093/jamia/ocaf233

Vancouver

Du X, Zhou Z, Wang Y, Chuang YW, Li Y, Yang R, et al. Testing and evaluation of generative large language models in electronic health record applications: a systematic review. Journal of the American Medical Informatics Association. 2026;33(3):743-753. doi:10.1093/jamia/ocaf233

BibTeX
@article{Du2026,
  author = {Du, Xinsong and Zhou, Zhengyang and Wang, Yifei and Chuang, Ya-Wen and Li, Yiming and Yang, Richard and Zhang, Wenyu and Wang, Xinyi and Chen, Xinyu and Guan, Hao and Lian, John and Hong, Pengyu and Bates, David W and Zhou, Li},
  title = {Testing and evaluation of generative large language models in electronic health record applications: a systematic review},
  journal = {Journal of the American Medical Informatics Association},
  year = {2026},
  volume = {33},
  number = {3},
  pages = {743--753},
  doi = {10.1093/jamia/ocaf233}
}
Kostenlos starten →