EUVIMEDEUVIMED entdecken
Open Access

Performance Drift in Machine Learning Models for Cardiac Surgery Risk Prediction: Retrospective Analysis

Tim Dong, Shubhra Sinha, Ben Zhai, Daniel Fudulu, Jeremy Chan, Pradeep Narayan, Andy Judge, Massimo Caputo, Arnaldo Dimagli, Umberto Benedetto, Gianni D Angelini

JMIRx Med · 12.06.2024 · 5 · e45973-e45973

DOI: 10.2196/45973 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIRx Med, online erschienen am 12.06.2024. Verfasst von Tim Dong und 10 weiteren Autor:innen. Beteiligte Einrichtung: Bristol Heart Institute, Translational Health Sciences, University of Bristol, Bristol, United Kingdom. Die Arbeit wurde bisher 17-mal zitiert (Crossref) und nennt 63 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: British Heart Foundation.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

The Society of Thoracic Surgeons and European System for Cardiac Operative Risk Evaluation (EuroSCORE) II risk scores are the most commonly used risk prediction models for in-hospital mortality after adult cardiac surgery. However, they are prone to miscalibration over time and poor generalization across data sets; thus, their use remains controversial. Despite increased interest, a gap in understanding the effect of data set drift on the performance of machine learning (ML) over time remains a barrier to its wider use in clinical practice. Data set drift occurs when an ML system underperforms because of a mismatch between the data it was developed from and the data on which it is deployed.

Objective

In this study, we analyzed the extent of performance drift using models built on a large UK cardiac surgery database. The objectives were to (1) rank and assess the extent of performance drift in cardiac surgery risk ML models over time and (2) investigate any potential influence of data set drift and variable importance drift on performance drift.

Methods

We conducted a retrospective analysis of prospectively, routinely gathered data on adult patients undergoing cardiac surgery in the United Kingdom between 2012 and 2019. We temporally split the data 70:30 into a training and validation set and a holdout set. Five novel ML mortality prediction models were developed and assessed, along with EuroSCORE II, for relationships between and within variable importance drift, performance drift, and actual data set drift. Performance was assessed using a consensus metric.

Results

A total of 227,087 adults underwent cardiac surgery during the study period, with a mortality rate of 2.76% (n=6258). There was strong evidence of a decrease in overall performance across all models ( P

Schlagwörter

Stichwörter der Autor:innen

United Kingdomadultartificial intelligencecardiaccardiac surgerycardiologydatadata set driftheartmachine learningmodelmortalitynational data setoperative mortalityperformance

Autor:innen und Institutionen

  • Bristol Heart Institute, Translational Health Sciences, University of Bristol, Bristol, United Kingdom.
  • School of Computing Science, Northumbria University, Newcastle upon Tyne, United Kingdom.
  • Department of Cardiac Surgery, Rabindranath Tagore International Institute of Cardiac Sciences, West Bengal, India.

Förderung

  • British Heart Foundation

Zitieren

APA

Dong, T., Sinha, S., Zhai, B., Fudulu, D., Chan, J., Narayan, P., Judge, A., Caputo, M., Dimagli, A., Benedetto, U., & Angelini, G. D. (2024). Performance Drift in Machine Learning Models for Cardiac Surgery Risk Prediction: Retrospective Analysis. JMIRx Med, 5, e45973-e45973. https://doi.org/10.2196/45973

Vancouver

Dong T, Sinha S, Zhai B, Fudulu D, Chan J, Narayan P, et al. Performance Drift in Machine Learning Models for Cardiac Surgery Risk Prediction: Retrospective Analysis. JMIRx Med. 2024;5:e45973-e45973. doi:10.2196/45973

BibTeX
@article{Dong2024,
  author = {Dong, Tim and Sinha, Shubhra and Zhai, Ben and Fudulu, Daniel and Chan, Jeremy and Narayan, Pradeep and Judge, Andy and Caputo, Massimo and Dimagli, Arnaldo and Benedetto, Umberto and Angelini, Gianni D},
  title = {Performance Drift in Machine Learning Models for Cardiac Surgery Risk Prediction: Retrospective Analysis},
  journal = {JMIRx Med},
  year = {2024},
  volume = {5},
  pages = {e45973--e45973},
  doi = {10.2196/45973}
}
Kostenlos starten →