EUVIMEDTutustu EUVIMEDiin
Open access · CC BY 4.0

Calibration plots for multistate risk prediction models

Alexander Pate iD, Matthew Sperrin iD, Richard D Riley, Niels Peek, Tjeerd Van Staa, Jamie C Sergeant, Mamas A Mamas, Gregory Y H Lip, Martin O'Flaherty, Michael Barrowman, Iain Buchan, Glen P Martin iD

Statistics in Medicine · 2024-06-30 · 43 (14) · 2830-2852

DOI: 10.1002/sim.10094 ↗

At a glance

Journal article in Statistics in Medicine, published online on 2024-05-08. Written by Alexander Pate and 11 co-authors. Institution: Centre for Health Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Healt…. Major MEDLINE topics: Diabetes Mellitus, Type 2, Models, Statistical, Computer Simulation. Cited 11 times so far (Crossref); 103 references. The full text is freely available (open access, CC BY 4.0). Funded by: Medical Research Council.

TiivistelmäSubjectsBibliografiset tiedotCiteRelated studies

Abstract (original language)

Introduction

There is currently no guidance on how to assess the calibration of multistate models used for risk prediction. We introduce several techniques that can be used to produce calibration plots for the transition probabilities of a multistate model, before assessing their performance in the presence of random and independent censoring through a simulation.

Methods

We studied pseudo‐values based on the Aalen‐Johansen estimator, binary logistic regression with inverse probability of censoring weights (BLR‐IPCW), and multinomial logistic regression with inverse probability of censoring weights (MLR‐IPCW). The MLR‐IPCW approach results in a calibration scatter plot, providing extra insight about the calibration. We simulated data with varying levels of censoring and evaluated the ability of each method to estimate the calibration curve for a set of predicted transition probabilities. We also developed and evaluated the calibration of a model predicting the incidence of cardiovascular disease, type 2 diabetes and chronic kidney disease among a cohort of patients derived from linked primary and secondary healthcare records.

Results

The pseudo‐value, BLR‐IPCW, and MLR‐IPCW approaches give unbiased estimates of the calibration curves under random censoring. These methods remained predominately unbiased in the presence of independent censoring, even if the censoring mechanism was strongly associated with the outcome, with bias concentrated in low‐density regions of predicted transition probabilities.

Conclusions

We recommend implementing either the pseudo‐value or BLR‐IPCW approaches to produce a calibration curve, combined with the MLR‐IPCW approach to produce a calibration scatter plot. The methods have been incorporated into the “calibmsm” R package available on CRAN.

Subjects

MeSH terms (MEDLINE)

Diabetes Mellitus, Type 2Models, StatisticalComputer SimulationHumansRisk AssessmentLogistic ModelsCalibrationCardiovascular DiseasesRenal Insufficiency, ChronicProbability

Author keywords

calibrationclinical predictionmodel validationmultistate modelrisk prediction

Authors and institutions

  • Centre for Health Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK.
  • NIHR Manchester Biomedical Research Centre, University of Manchester, Manchester, UK.
  • Institute of Applied Health Research, University of Birmingham, Birmingham, UK.
  • Centre for Epidemiology Versus Arthritis, Centre for Musculoskeletal Research, Manchester Academic Health Science Centre, University of Manchester, Manchester, UK.
  • Centre for Biostatistics, Manchester Academic Health Science Centre, University of Manchester, Manchester, UK.
  • Keele Cardiovascular Research Group, Keele University, Stoke-on-Trent, UK.

ORCID profiles: Alexander Pate, Matthew Sperrin, Glen P Martin

Funding

  • Medical Research Council

Cite

APA

Pate, A., Sperrin, M., Riley, R. D., Peek, N., Van Staa, T., Sergeant, J. C., Mamas, M. A., Lip, G. Y. H., O'Flaherty, M., Barrowman, M., Buchan, I., & Martin, G. P. (2024). Calibration plots for multistate risk prediction models. Statistics in Medicine, 43(14), 2830-2852. https://doi.org/10.1002/sim.10094

Vancouver

Pate A, Sperrin M, Riley RD, Peek N, Van Staa T, Sergeant JC, et al. Calibration plots for multistate risk prediction models. Statistics in Medicine. 2024;43(14):2830-2852. doi:10.1002/sim.10094

BibTeX
@article{Pate2024,
  author = {Pate, Alexander and Sperrin, Matthew and Riley, Richard D and Peek, Niels and Van Staa, Tjeerd and Sergeant, Jamie C and Mamas, Mamas A and Lip, Gregory Y H and O'Flaherty, Martin and Barrowman, Michael and Buchan, Iain and Martin, Glen P},
  title = {Calibration plots for multistate risk prediction models},
  journal = {Statistics in Medicine},
  year = {2024},
  volume = {43},
  number = {14},
  pages = {2830--2852},
  doi = {10.1002/sim.10094}
}
Start for free →