Beyond the Model: Practical Insights from Monitoring Predictive Models across Diverse Clinical Workflows
Applied Clinical Informatics · 01.05.2026 · 17 (03) · 384-391
DOI: 10.1055/a-2876-1059 ↗Kurz gefasst
Fachartikel in der Zeitschrift Applied Clinical Informatics, erschienen am 01.05.2026. Verfasst von John J Hanna und 6 weiteren Autor:innen. Beteiligte Einrichtung: ECU Health, Information Services, North Carolina, United States, Greenville. Hauptthemen laut MEDLINE: Workflow. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 9 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
To support the artificial intelligence (AI) lifecycle in an integrated academic health system, we implemented a modular monitoring system to oversee electronic health record (EHR) vendor-provided clinical predictive models. This case study describes the lessons learned using this modular system to support sustainable oversight of the deployed predictive models across their lifecycle. We developed a modular monitoring system using automated data pipelines refreshed daily to support longitudinal oversight of four clinical predictive models developed by our EHR vendor (Epic Systems). An interactive monitoring application was designed to bridge the technical–operational gap for stakeholders involved in AI lifecycle decisions. Interactive reports include automated and assisted threshold-independent and threshold-dependent performance measures, flag rates, and problem-specific metrics crafted to support operational decision-making. Through the lenses of people, process, and technology, we describe lessons learned from the real-world implementation of a monitoring system. Effective oversight required clearly defined dyadic ownership (technical and operational) and embedded monitoring activities within existing AI governance decision-making processes across all AI lifecycle stages. Monitoring extended beyond performance drift to include alert burden, workflow alignment, and problem-level signals. Modularity enabled rapid adaptation as models and workflows evolved. Silent alert simulations proved valuable for selecting preimplementation thresholds that aligned with clinical capacity and workflows but were limited in predicting postimplementation performance when the models faced real-world human–alert interactions. Implementing a modular, governance-aligned monitoring system enabled sustained oversight of vendor-provided clinical predictive models and shifted monitoring beyond traditional technical performance metrics toward user-centric and operationally meaningful measures. These findings highlight the importance of integrated monitoring infrastructure as a core component of responsible AI governance in clinical settings.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- ECU Health, Information Services, North Carolina, United States, Greenville.
- East Carolina University Brody School of Medicine, Department of Internal Medicine, North Carolina, United States, Greenville.
- University of Texas Southwestern, Department of Internal Medicine, Texas, United States, Dallas.
- University of Texas Southwestern, Clinical Informatics Center, Texas, United States, Dallas.
- Medical University of South Carolina, College of Health Professions, Department of Healthcare Leadership and Management, South Carolina, United States, Charleston.
- East Carolina University Brody School of Medicine, Department of Emergency Medicine, North Carolina, United States, Greenville.
ORCID-Profile: Lauren N Cooper
Zitieren
APA
Hanna, J. J., Dennis, C. R., Johnson, A. O., Cooper, L. N., Evans, C. S., Lehmann, C. U., & Medford, R. J. (2026). Beyond the Model: Practical Insights from Monitoring Predictive Models across Diverse Clinical Workflows. Applied Clinical Informatics, 17(03), 384-391. https://doi.org/10.1055/a-2876-1059
Vancouver
Hanna JJ, Dennis CR, Johnson AO, Cooper LN, Evans CS, Lehmann CU, et al. Beyond the Model: Practical Insights from Monitoring Predictive Models across Diverse Clinical Workflows. Applied Clinical Informatics. 2026;17(03):384-391. doi:10.1055/a-2876-1059
BibTeX
@article{Hanna2026,
author = {Hanna, John J and Dennis, Christopher R and Johnson, Andrew O and Cooper, Lauren N and Evans, Christopher S and Lehmann, Christoph Ulrich and Medford, Richard J},
title = {Beyond the Model: Practical Insights from Monitoring Predictive Models across Diverse Clinical Workflows},
journal = {Applied Clinical Informatics},
year = {2026},
volume = {17},
number = {03},
pages = {384--391},
doi = {10.1055/a-2876-1059}
}
EUVIMED