Information extraction from clinical notes: are we ready to switch to large language models?
Journal of the American Medical Informatics Association · 01.03.2026 · 33 (3) · 553-562
DOI: 10.1093/jamia/ocaf213 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 14.01.2026. Verfasst von Yan Hu und 12 weiteren Autor:innen. Beteiligte Einrichtung: McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX 77030, United States. Hauptthemen laut MEDLINE: Large Language Models, Electronic Health Records, Information Storage and Retrieval, Natural Language Processing. Die Arbeit wurde bisher 33-mal zitiert (Crossref) und nennt 52 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: NIA NIH HHS, National Institute on Aging.
Abstract (Originalsprache)
Objectives
To assess the performance, generalizability, and computational efficiency of instruction-tuned Large Language Model Meta AI (LLaMA)-2 and LLaMA-3 models compared to bidirectional encoder representations from transformers (BERT) for clinical information extraction (IE) tasks, specifically named entity recognition (NER) and relation extraction (RE). Materials and Methods We developed a comprehensive annotated corpus of 1588 clinical notes from 4 data sources—UT Physicians (UTP) (1342 notes), Transcribed Medical Transcription Sample Reports and Examples (MTSamples) (146), Medical Information Mart for Intensive Care (MIMIC)-III (50), and Informatics for Integrating Biology and the Bedside (i2b2) (50), capturing 4 clinical entities (problems, tests, medications, other treatments) and 16 modifiers (eg, negation, certainty). Large Language Model Meta AI-2 and LLaMA-3 were instruction-tuned for clinical NER and RE, and their performance was benchmarked against BERT.
Results
Large Language Model Meta AI models consistently outperformed BERT across datasets. In data-rich settings (eg, UTP), LLaMA achieved marginal gains (approximately 1% improvement for NER and 1.5%-3.7% for RE). Under limited data conditions (eg, MTSamples, MIMIC-III) and on the unseen i2b2 dataset, LLaMA-3-70B improved F1 scores by over 7% for NER and 4% for RE. However, performance gains came with increased computational costs, with LLaMA models requiring more memory and Graphics Processing Unit (GPU) hours and running up to 28 times slower than BERT.
Discussion
While LLaMA models offer enhanced performance, their higher computational demands and slower throughput highlight the need to balance performance with practical resource constraints. Application-specific considerations are essential when choosing between LLMs and BERT for clinical IE.
Conclusion
Instruction-tuned LLaMA models show promise for clinical NER and RE tasks. However, the tradeoff between improved performance and increased computational cost must be carefully evaluated. We release our Kiwi package (https://kiwi.clinicalnlp.org/) to facilitate the application of both LLaMA and BERT models in clinical IE applications.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX 77030, United States.
- Department of Biomedical Informatics and Data Science, Yale School of Medicine, Yale University, New Haven, CT 06510, United States.
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37203, United States.
ORCID-Profile: Yan Hu, Vipina K Keloth, Cathy Shyr, Xiaoqian Jiang, Hua Xu
Förderung
- NIA NIH HHS
- National Institute on Aging
Zitieren
APA
Hu, Y., Zuo, X., Zhou, Y., Peng, X., Huang, J., Keloth, V. K., Zhang, V. J., Weng, R. L., Shyr, C., Chen, Q., Jiang, X., Roberts, K. E., & Xu, H. (2026). Information extraction from clinical notes: are we ready to switch to large language models?. Journal of the American Medical Informatics Association, 33(3), 553-562. https://doi.org/10.1093/jamia/ocaf213
Vancouver
Hu Y, Zuo X, Zhou Y, Peng X, Huang J, Keloth VK, et al. Information extraction from clinical notes: are we ready to switch to large language models?. Journal of the American Medical Informatics Association. 2026;33(3):553-562. doi:10.1093/jamia/ocaf213
BibTeX
@article{Hu2026,
author = {Hu, Yan and Zuo, Xu and Zhou, Yujia and Peng, Xueqing and Huang, Jimin and Keloth, Vipina K and Zhang, Vincent J and Weng, Ruey-Ling and Shyr, Cathy and Chen, Qingyu and Jiang, Xiaoqian and Roberts, Kirk E and Xu, Hua},
title = {Information extraction from clinical notes: are we ready to switch to large language models?},
journal = {Journal of the American Medical Informatics Association},
year = {2026},
volume = {33},
number = {3},
pages = {553--562},
doi = {10.1093/jamia/ocaf213}
}
EUVIMED