EUVIMEDScopri EUVIMED
Open access

Performance Drift in Machine Learning Models for Cardiac Surgery Risk Prediction: Retrospective Analysis

Tim Dong, Shubhra Sinha, Ben Zhai, Daniel Fudulu, Jeremy Chan, Pradeep Narayan, Andy Judge, Massimo Caputo, Arnaldo Dimagli, Umberto Benedetto, Gianni D Angelini

JMIRx Med · 2024-06-12 · 5 · e45973-e45973

DOI: 10.2196/45973 ↗

At a glance

Journal article in JMIRx Med, published online on 2024-06-12. Written by Tim Dong and 10 co-authors. Institution: Bristol Heart Institute, Translational Health Sciences, University of Bristol, Bristol, United Kingdom. Cited 17 times so far (Crossref); 63 references. The full text is freely available in PubMed Central. Funded by: British Heart Foundation.

AbstractSubjectsDati bibliograficiCiteRelated studies

Abstract (original language)

Background

The Society of Thoracic Surgeons and European System for Cardiac Operative Risk Evaluation (EuroSCORE) II risk scores are the most commonly used risk prediction models for in-hospital mortality after adult cardiac surgery. However, they are prone to miscalibration over time and poor generalization across data sets; thus, their use remains controversial. Despite increased interest, a gap in understanding the effect of data set drift on the performance of machine learning (ML) over time remains a barrier to its wider use in clinical practice. Data set drift occurs when an ML system underperforms because of a mismatch between the data it was developed from and the data on which it is deployed.

Objective

In this study, we analyzed the extent of performance drift using models built on a large UK cardiac surgery database. The objectives were to (1) rank and assess the extent of performance drift in cardiac surgery risk ML models over time and (2) investigate any potential influence of data set drift and variable importance drift on performance drift.

Methods

We conducted a retrospective analysis of prospectively, routinely gathered data on adult patients undergoing cardiac surgery in the United Kingdom between 2012 and 2019. We temporally split the data 70:30 into a training and validation set and a holdout set. Five novel ML mortality prediction models were developed and assessed, along with EuroSCORE II, for relationships between and within variable importance drift, performance drift, and actual data set drift. Performance was assessed using a consensus metric.

Results

A total of 227,087 adults underwent cardiac surgery during the study period, with a mortality rate of 2.76% (n=6258). There was strong evidence of a decrease in overall performance across all models ( P

Subjects

Author keywords

United Kingdomadultartificial intelligencecardiaccardiac surgerycardiologydatadata set driftheartmachine learningmodelmortalitynational data setoperative mortalityperformance

Authors and institutions

  • Bristol Heart Institute, Translational Health Sciences, University of Bristol, Bristol, United Kingdom.
  • School of Computing Science, Northumbria University, Newcastle upon Tyne, United Kingdom.
  • Department of Cardiac Surgery, Rabindranath Tagore International Institute of Cardiac Sciences, West Bengal, India.

Funding

  • British Heart Foundation

Cite

APA

Dong, T., Sinha, S., Zhai, B., Fudulu, D., Chan, J., Narayan, P., Judge, A., Caputo, M., Dimagli, A., Benedetto, U., & Angelini, G. D. (2024). Performance Drift in Machine Learning Models for Cardiac Surgery Risk Prediction: Retrospective Analysis. JMIRx Med, 5, e45973-e45973. https://doi.org/10.2196/45973

Vancouver

Dong T, Sinha S, Zhai B, Fudulu D, Chan J, Narayan P, et al. Performance Drift in Machine Learning Models for Cardiac Surgery Risk Prediction: Retrospective Analysis. JMIRx Med. 2024;5:e45973-e45973. doi:10.2196/45973

BibTeX
@article{Dong2024,
  author = {Dong, Tim and Sinha, Shubhra and Zhai, Ben and Fudulu, Daniel and Chan, Jeremy and Narayan, Pradeep and Judge, Andy and Caputo, Massimo and Dimagli, Arnaldo and Benedetto, Umberto and Angelini, Gianni D},
  title = {Performance Drift in Machine Learning Models for Cardiac Surgery Risk Prediction: Retrospective Analysis},
  journal = {JMIRx Med},
  year = {2024},
  volume = {5},
  pages = {e45973--e45973},
  doi = {10.2196/45973}
}
Start for free →