EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access

Benchmarking information extraction of physical activity from electronic health record with large language models: an natural language processing pipeline and comparative evaluation

Han Yang iD, Zhongran Niu iD, Mingchen Li, Huixue Zhou iD, Yongkang Xiao iD, Sicheng Zhou, Zaifu Zhan, Ying Liu, Shiqin Liu, Christopher J Tignanelli iD, Genevieve B Melton iD, Rui Zhang

Journal of the American Medical Informatics Association · 2026

DOI: 10.1093/jamia/ocag101 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift Journal of the American Medical Informatics Association, 2026. Verfasst von Han Yang und 11 weiteren Autor:innen. Beteiligte Einrichtung: Institute for Health Informatics, University of Minnesota, Minneapolis, MN 55455, United States. Hauptthemen laut MEDLINE: Large Language Models, Electronic Health Records, Natural Language Processing, Exercise. Die Arbeit nennt 55 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: National Institutes of Health's National Center for Complementary and Integrative Health, NIA NIH HHS, NCI NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

We aimed to develop a data model and a natural language processing (NLP) pipeline for representing physical activity (PA) in Electronic Health Records (EHRs), and to evaluate transformer- and Large Language Model (LLM)-based classifiers for sentence-level PA attribute classification. Materials and Methods We analyzed PA documentation across three patient cohorts (cancer, COVID, and Alzheimer's disease) using structured and unstructured EHR data. A conceptual schema was developed to represent PA and its linguistic attributes. Five BERT models and three modern LLMs (Llama3-8B, MedAlpaca-13B, and PMC-Llama-13B) were evaluated for classifying PA attributes (binary status, negation, exclusion, and an eleven-class Category) on pre-extracted PA-related sentences.

Results

Clinical notes were a richer source of PA information than structured ICD or SDoH data. On binary tasks, the best BERT model reached F1 0.619 (Exclusion); with Supervised Fine-Tuning (SFT), Llama3-8B reached F1 0.689 (Exclusion). On the 11-class Category task, performance was modest (best macro-F1 0.262, ROC-AUC 0.803, by Llama3-8B).

Discussion

In-Context Learning (ICL) was highly variable: while Llama3-8B-ICL achieved the best ROC-AUC on Category, the domain-specific MedAlpaca-13B and PMC-LLaMA-13B essentially failed. These results, together with sparsely represented PA elements (Amount, Frequency, Assessment) and the absence of a downstream evaluation, position this work as an initial proof of feasibility, with supervised domain adaptation still required for reliable clinical PA extraction.

Conclusion

We contribute a PA data model, annotation schema, and a working NLP pipeline with a BERT/LLM benchmark for sentence-level PA attribute classification. The pipeline supports future end-to-end PA extraction and downstream applications such as phenotyping, risk prediction, and cohort identification.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsElectronic Health RecordsNatural Language ProcessingExerciseHumansBenchmarking

Stichwörter der Autor:innen

BERTLlama3electronic health recordinformation extractionlarge language modelsphysical activity

Autor:innen und Institutionen

  • Institute for Health Informatics, University of Minnesota, Minneapolis, MN 55455, United States.
  • Division of Computational Health Sciences, Department of Surgery, University of Minnesota, Minneapolis, MN 55455, United States.
  • Center for Transportation Studies, University of Minnesota, Minneapolis, MN 55455, United States.
  • Division of Critical Care and Acute Care Surgery, Department of Surgery, University of Minnesota, Minneapolis, MN 55455, United States.
  • Center for Learning Health System Sciences, University of Minnesota, Minneapolis, MN 55455, United States.
  • Division of Colon and Rectal Surgery, Department of Surgery, University of Minnesota, Minneapolis, MN 55455, United States.

ORCID-Profile: Han Yang, Zhongran Niu, Huixue Zhou, Yongkang Xiao, Christopher J Tignanelli, Genevieve B Melton

Förderung

  • National Institutes of Health's National Center for Complementary and Integrative Health
  • NIA NIH HHS
  • NCI NIH HHS
  • NIDDK NIH HHS
  • NIMHD NIH HHS
  • FDA HHS
  • University of Minnesota Clinical and Translational Science Institute (CTSI)
  • NCATS NIH HHS

Zitieren

APA

Yang, H., Niu, Z., Li, M., Zhou, H., Xiao, Y., Zhou, S., Zhan, Z., Liu, Y., Liu, S., Tignanelli, C. J., Melton, G. B., & Zhang, R. (2026). Benchmarking information extraction of physical activity from electronic health record with large language models: an natural language processing pipeline and comparative evaluation. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocag101

Vancouver

Yang H, Niu Z, Li M, Zhou H, Xiao Y, Zhou S, et al. Benchmarking information extraction of physical activity from electronic health record with large language models: an natural language processing pipeline and comparative evaluation. Journal of the American Medical Informatics Association. 2026. doi:10.1093/jamia/ocag101

BibTeX
@article{Yang2026,
  author = {Yang, Han and Niu, Zhongran and Li, Mingchen and Zhou, Huixue and Xiao, Yongkang and Zhou, Sicheng and Zhan, Zaifu and Liu, Ying and Liu, Shiqin and Tignanelli, Christopher J and Melton, Genevieve B and Zhang, Rui},
  title = {Benchmarking information extraction of physical activity from electronic health record with large language models: an natural language processing pipeline and comparative evaluation},
  journal = {Journal of the American Medical Informatics Association},
  year = {2026},
  doi = {10.1093/jamia/ocag101}
}
Kostenlos starten →