EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Beyond AUROC: linking predictions to clinical workflows in deployed decision support systems

Georgy Kopanitsa, Dmitry Kurapeev, Nikolai Shelukhanov, Evgeny Karpulevich, Yury Markin, Oleg Metsker, Alexey Yakovlev

Frontiers in Digital Health · 18.11.2026 · 8

DOI: 10.3389/fdgth.2026.1960268 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Frontiers in Digital Health, online erschienen am 18.11.2026. Verfasst von Georgy Kopanitsa und 6 weiteren Autor:innen. Die Arbeit nennt 33 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Clinical decision support systems (CDSS) require an assessment of predictive performance and the pathway supplying inputs, routing alerts, and recording workflow responses. Aggregate monitoring may not identify where predictions fail, while recorded digital interactions do not establish clinical appropriateness or patient benefit.

Objective

To determine whether linking records for each prediction across input data, alert decisions, delivery, clinician interaction, and outcomes identifies reliability signals missed by model performance alone.

Methods

We retrospectively evaluated deidentified EHR records and operational logs for four CDSS workflows: sepsis warning, inpatient deterioration, 30-day readmission, and outpatient laboratory-risk follow-up. The systems used three tree-based classifiers and one L2-regularized logistic regression. Data included 4,200 patients, 6,008 encounters, 9,402 episodes, 17,768 predictions, and 2,950 routing records. Measures before delivery covered silent validation, early deployment, and mature deployment; delivery and interaction outcomes were restricted to clinician-facing phases. Predictions were linked through input availability and timing, risk estimation, alert decisions, routing or delivery, interactions, and structured outcomes. Record review included 427 episodes.

Results

AUROC changes showed no consistent direction across systems (0.692–0.789); System C declined from 0.774 during silent validation to 0.692 during mature deployment. Calibration slopes ranged from 0.511 to 0.823, calibration-in-the-large from −0.633 to 0.022, and observed-to-expected ratios from 0.674 to 1.013. Early deployment had lower feature availability and longer ingestion latency. Across combinations of workflow and phase, 67.2%–87.8% of predictions had at least one unavailable feature, 21.5%–40.6% had an ingestion event after prediction, and routing or delivery failure was 2.1%–6.3%. During clinical deployment, 95.6% of production alerts were delivered; opening, acknowledgment, override, and recorded action rates were 63.8%–75.2%, 48.0%–59.0%, 10.1%–18.5%, and 17.3%–29.0%. None of the 160 subgroup contrasts remained supported after multiplicity adjustment.

Conclusion

Linking records for each prediction located reliability signals across input data, routing or delivery, and recorded responses that were not apparent from discrimination or performance at the operating threshold alone. Evaluation of the complete workflow complements rather than replaces discrimination, calibration, reference outcome assessment, and clinical utility evaluation. This retrospective single-institution study did not estimate causal effects or patient benefit; its estimates are local, and the monitoring structure is not a validated universal framework.

Zitieren

APA

Kopanitsa, G., Kurapeev, D., Shelukhanov, N., Karpulevich, E., Markin, Y., Metsker, O., & Yakovlev, A. (2026). Beyond AUROC: linking predictions to clinical workflows in deployed decision support systems. Frontiers in Digital Health, 8. https://doi.org/10.3389/fdgth.2026.1960268

Vancouver

Kopanitsa G, Kurapeev D, Shelukhanov N, Karpulevich E, Markin Y, Metsker O, et al. Beyond AUROC: linking predictions to clinical workflows in deployed decision support systems. Frontiers in Digital Health. 2026;8. doi:10.3389/fdgth.2026.1960268

BibTeX
@article{Kopanitsa2026,
  author = {Kopanitsa, Georgy and Kurapeev, Dmitry and Shelukhanov, Nikolai and Karpulevich, Evgeny and Markin, Yury and Metsker, Oleg and Yakovlev, Alexey},
  title = {Beyond AUROC: linking predictions to clinical workflows in deployed decision support systems},
  journal = {Frontiers in Digital Health},
  year = {2026},
  volume = {8},
  doi = {10.3389/fdgth.2026.1960268}
}
Kostenlos starten →