Electronic health record activity changes around new decision support implementation: monitoring using audit logs and topic modeling
JAMIA Open · 03.07.2025 · 8 (4)
DOI: 10.1093/jamiaopen/ooaf050 ↗Kurz gefasst
Fachartikel in der Zeitschrift JAMIA Open, online erschienen am 15.07.2025. Verfasst von Jinying Chen und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Medicine/Section of Preventive Medicine and Epidemiology, Boston University Chobanian & Avedisian School of Medicine, Boston, MA 02118,…. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 84 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: NCATS NIH HHS, National Cancer Institute, National Institutes of Health.
Abstract (Originalsprache)
Objectives
To develop and test a novel machine learning approach for monitoring impact of computerized clinical decision support (CDS) tools on clinicians’ electronic health record (EHR) activities. Materials and Methods Our CDS monitoring approach leverages topic modeling, a latent-variable statistical machine learning method, to infer health providers’ EHR activities from EHR audit logs. We applied this approach to monitor the impact of a tobacco cessation support CDS tool newly implemented in 5 cancer clinics (2018-2021). We trained the topic model on EHR audit log data from 3445 encounters (pre-CDS-implementation: 1734, post-CDS-implementation: 1711) for patients with active smoking status. The number of topics was automatically determined based on within-topic coherence and across-topic divergence, and the identified topics were assigned clinically relevant EHR activity labels by 4 domain experts.
Results
The topic model identified 2 distinct activities focusing on CDS (act on CDS, bypass/postpone CDS), 2 activities related to CDS (review patient records and address alerts, use note templates and acknowledge the completion of CDS), 6 related to accessing (access patient station) and reviewing patient data (external records, synopsis data, snapshot of patient data, problem list/diagnosis/notes, treatment plan), and 4 related to modifying EHR (modify diagnosis/problem lists, document visit with record review, perform administrative activities for visit and billing, and document follow-up care plan). Comparing matched 1-hour after-check-in windows post-implementation (n = 841) versus pre-implementation (n = 841) of CDS, the mean prevalence (expressed as proportions out of 1.0) of providers’ EHR-use activity increased on CDS-focused activities (0.073, 95% CI, 0.066-0.079) and CDS-related activities (0.098, 95% CI, 0.089-0.106) and decreased on modifying EHR (−0.113, 95% CI, −0.124 to −0.102) and reviewing patient data (−0.058, 95% CI, −0.072 to −0.044).
Discussion
Our topic model-based CDS monitoring approach can identify shifts in prevalence of EHR-use activities pre-implementation versus post-implementation. This approach can be applied to detect unintended changes in EHR activities on a large population scale following CDS implementation, providing valuable insights to guide focused qualitative investigations for CDS improvement or de-implementation.
Conclusion
Our approach offers a scalable, data-driven framework for evaluating the real-world impact of EHR-embedded CDS tools. Built on a generic machine learning framework, this approach could be adapted to explore impact of other healthcare quality improvement strategies using EHR-integrated CDS interventions.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Medicine/Section of Preventive Medicine and Epidemiology, Boston University Chobanian & Avedisian School of Medicine, Boston, MA 02118, United States.
- Data Science Core, Boston University Chobanian & Avedisian School of Medicine, Boston, MA 02118, United States.
- Department of Internal Medicine/Section of General Internal Medicine, Wake Forest University School of Medicine, Winston Salem, NC 27101, United States.
- Department of Population and Quantitative Health Sciences, University of Massachusetts Chan Medical School, Worcester, MA 01655, United States.
- Center for Healthcare Organization and Implementation Research (CHOIR), VA Bedford Healthcare System, Bedford, MA 01730, United States.
- Department of Implementation Science, Division of Public Health Sciences, Wake Forest University School of Medicine, Winston Salem, NC 27101, United States.
ORCID-Profile: Jinying Chen, Sarah L Cutrona, Ajay Dharod, Adam Moses, Aaron Bridges, Brian Ostasiewski, Kristie L Foley, Thomas K Houston
Förderung
- NCATS NIH HHS
- National Cancer Institute
- National Institutes of Health
Zitieren
APA
Chen, J., Cutrona, S. L., Dharod, A., Moses, A., Bridges, A., Ostasiewski, B., Foley, K. L., Houston, T. K., & iDAPT (Implementation & Informatics Developing Adaptable Processes and Technologies for Cancer Control) Implementation Science Center for Cancer Control (2025). Electronic health record activity changes around new decision support implementation: monitoring using audit logs and topic modeling. JAMIA Open, 8(4). https://doi.org/10.1093/jamiaopen/ooaf050
Vancouver
Chen J, Cutrona SL, Dharod A, Moses A, Bridges A, Ostasiewski B, et al. Electronic health record activity changes around new decision support implementation: monitoring using audit logs and topic modeling. JAMIA Open. 2025;8(4). doi:10.1093/jamiaopen/ooaf050
BibTeX
@article{Chen2025,
author = {Chen, Jinying and Cutrona, Sarah L and Dharod, Ajay and Moses, Adam and Bridges, Aaron and Ostasiewski, Brian and Foley, Kristie L and Houston, Thomas K and iDAPT (Implementation & Informatics Developing Adaptable Processes and Technologies for Cancer Control) Implementation Science Center for Cancer Control, iDAPT (Implementation & Informatics Developing Adaptable Processes and Technologies for Cancer Control) Implementation Science Center for Cancer Control},
title = {Electronic health record activity changes around new decision support implementation: monitoring using audit logs and topic modeling},
journal = {JAMIA Open},
year = {2025},
volume = {8},
number = {4},
doi = {10.1093/jamiaopen/ooaf050}
}
EUVIMED