EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Stability of clinical prediction models developed using statistical or machine learning methods

Richard D Riley iD, Gary S Collins iD

Biometrical Journal · 2023-12 · 65 (8)

DOI: 10.1002/bimj.202200302 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Biometrical Journal, online erschienen am 19.07.2023. Verfasst von Richard D Riley und 1 weiteren Autor:innen. Beteiligte Einrichtung: Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK. Hauptthemen laut MEDLINE: Models, Statistical, Machine Learning. Die Arbeit wurde bisher 231-mal zitiert (Crossref) und nennt 61 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Cancer Research UK, Medical Research Council.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Clinical prediction models estimate an individual's risk of a particular health outcome. A developed model is a consequence of the development dataset and model‐building strategy, including the sample size, number of predictors, and analysis method (e.g., regression or machine learning). We raise the concern that many models are developed using small datasets that lead to instability in the model and its predictions (estimated risks). We define four levels of model stability in estimated risks moving from the overall mean to the individual level. Through simulation and case studies of statistical and machine learning approaches, we show instability in a model's estimated risks is often considerable, and ultimately manifests itself as miscalibration of predictions in new data. Therefore, we recommend researchers always examine instability at the model development stage and propose instability plots and measures to do so. This entails repeating the model‐building steps (those used to develop the original prediction model) in each of multiple (e.g., 1000) bootstrap samples, to produce multiple bootstrap models, and deriving (i) a prediction instability plot of bootstrap model versus original model predictions; (ii) the mean absolute prediction error (mean absolute difference between individuals’ original and bootstrap model predictions), and (iii) calibration, classification, and decision curve instability plots of bootstrap models applied in the original sample. A case study illustrates how these instability assessments help reassure (or not) whether model predictions are likely to be reliable (or not), while informing a model's critical appraisal (risk of bias rating), fairness, and further validation requirements.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Models, StatisticalMachine LearningHumansPrognosisComputer Simulation

Stichwörter der Autor:innen

calibrationfairnessprediction modelstabilityuncertainty

Autor:innen und Institutionen

  • Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK.
  • Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK.

ORCID-Profile: Richard D Riley, Gary S Collins

Förderung

  • Cancer Research UK
  • Medical Research Council

Zitieren

APA

Riley, R. D., & Collins, G. S. (2023). Stability of clinical prediction models developed using statistical or machine learning methods. Biometrical Journal, 65(8). https://doi.org/10.1002/bimj.202200302

Vancouver

Riley RD, Collins GS. Stability of clinical prediction models developed using statistical or machine learning methods. Biometrical Journal. 2023;65(8). doi:10.1002/bimj.202200302

BibTeX
@article{Riley2023,
  author = {Riley, Richard D and Collins, Gary S},
  title = {Stability of clinical prediction models developed using statistical or machine learning methods},
  journal = {Biometrical Journal},
  year = {2023},
  volume = {65},
  number = {8},
  doi = {10.1002/bimj.202200302}
}
Kostenlos starten →