Benchmarking information extraction of physical activity from electronic health record with large language models: an natural language processing pipeline and comparative evaluation
Journal of the American Medical Informatics Association · 2026
DOI: 10.1093/jamia/ocag101 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Journal of the American Medical Informatics Association, 2026. Verfasst von Han Yang und 11 weiteren Autor:innen. Beteiligte Einrichtung: Institute for Health Informatics, University of Minnesota, Minneapolis, MN 55455, United States. Hauptthemen laut MEDLINE: Large Language Models, Electronic Health Records, Natural Language Processing, Exercise. Die Arbeit nennt 55 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: National Institutes of Health's National Center for Complementary and Integrative Health, NIA NIH HHS, NCI NIH HHS.
Abstract (Originalsprache)
Objectives
We aimed to develop a data model and a natural language processing (NLP) pipeline for representing physical activity (PA) in Electronic Health Records (EHRs), and to evaluate transformer- and Large Language Model (LLM)-based classifiers for sentence-level PA attribute classification. Materials and Methods We analyzed PA documentation across three patient cohorts (cancer, COVID, and Alzheimer's disease) using structured and unstructured EHR data. A conceptual schema was developed to represent PA and its linguistic attributes. Five BERT models and three modern LLMs (Llama3-8B, MedAlpaca-13B, and PMC-Llama-13B) were evaluated for classifying PA attributes (binary status, negation, exclusion, and an eleven-class Category) on pre-extracted PA-related sentences.
Results
Clinical notes were a richer source of PA information than structured ICD or SDoH data. On binary tasks, the best BERT model reached F1 0.619 (Exclusion); with Supervised Fine-Tuning (SFT), Llama3-8B reached F1 0.689 (Exclusion). On the 11-class Category task, performance was modest (best macro-F1 0.262, ROC-AUC 0.803, by Llama3-8B).
Discussion
In-Context Learning (ICL) was highly variable: while Llama3-8B-ICL achieved the best ROC-AUC on Category, the domain-specific MedAlpaca-13B and PMC-LLaMA-13B essentially failed. These results, together with sparsely represented PA elements (Amount, Frequency, Assessment) and the absence of a downstream evaluation, position this work as an initial proof of feasibility, with supervised domain adaptation still required for reliable clinical PA extraction.
Conclusion
We contribute a PA data model, annotation schema, and a working NLP pipeline with a BERT/LLM benchmark for sentence-level PA attribute classification. The pipeline supports future end-to-end PA extraction and downstream applications such as phenotyping, risk prediction, and cohort identification.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Institute for Health Informatics, University of Minnesota, Minneapolis, MN 55455, United States.
- Division of Computational Health Sciences, Department of Surgery, University of Minnesota, Minneapolis, MN 55455, United States.
- Center for Transportation Studies, University of Minnesota, Minneapolis, MN 55455, United States.
- Division of Critical Care and Acute Care Surgery, Department of Surgery, University of Minnesota, Minneapolis, MN 55455, United States.
- Center for Learning Health System Sciences, University of Minnesota, Minneapolis, MN 55455, United States.
- Division of Colon and Rectal Surgery, Department of Surgery, University of Minnesota, Minneapolis, MN 55455, United States.
ORCID-Profile: Han Yang, Zhongran Niu, Huixue Zhou, Yongkang Xiao, Christopher J Tignanelli, Genevieve B Melton
Förderung
- National Institutes of Health's National Center for Complementary and Integrative Health
- NIA NIH HHS
- NCI NIH HHS
- NIDDK NIH HHS
- NIMHD NIH HHS
- FDA HHS
- University of Minnesota Clinical and Translational Science Institute (CTSI)
- NCATS NIH HHS
Zitieren
APA
Yang, H., Niu, Z., Li, M., Zhou, H., Xiao, Y., Zhou, S., Zhan, Z., Liu, Y., Liu, S., Tignanelli, C. J., Melton, G. B., & Zhang, R. (2026). Benchmarking information extraction of physical activity from electronic health record with large language models: an natural language processing pipeline and comparative evaluation. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocag101
Vancouver
Yang H, Niu Z, Li M, Zhou H, Xiao Y, Zhou S, et al. Benchmarking information extraction of physical activity from electronic health record with large language models: an natural language processing pipeline and comparative evaluation. Journal of the American Medical Informatics Association. 2026. doi:10.1093/jamia/ocag101
BibTeX
@article{Yang2026,
author = {Yang, Han and Niu, Zhongran and Li, Mingchen and Zhou, Huixue and Xiao, Yongkang and Zhou, Sicheng and Zhan, Zaifu and Liu, Ying and Liu, Shiqin and Tignanelli, Christopher J and Melton, Genevieve B and Zhang, Rui},
title = {Benchmarking information extraction of physical activity from electronic health record with large language models: an natural language processing pipeline and comparative evaluation},
journal = {Journal of the American Medical Informatics Association},
year = {2026},
doi = {10.1093/jamia/ocag101}
}
EUVIMED