Calibration plots for multistate risk prediction models
Statistics in Medicine · 2024-06-30 · 43 (14) · 2830-2852
DOI: 10.1002/sim.10094 ↗At a glance
Journal article in Statistics in Medicine, published online on 2024-05-08. Written by Alexander Pate and 11 co-authors. Institution: Centre for Health Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Healt…. Major MEDLINE topics: Diabetes Mellitus, Type 2, Models, Statistical, Computer Simulation. Cited 11 times so far (Crossref); 103 references. The full text is freely available (open access, CC BY 4.0). Funded by: Medical Research Council.
Abstract (original language)
Introduction
There is currently no guidance on how to assess the calibration of multistate models used for risk prediction. We introduce several techniques that can be used to produce calibration plots for the transition probabilities of a multistate model, before assessing their performance in the presence of random and independent censoring through a simulation.
Methods
We studied pseudo‐values based on the Aalen‐Johansen estimator, binary logistic regression with inverse probability of censoring weights (BLR‐IPCW), and multinomial logistic regression with inverse probability of censoring weights (MLR‐IPCW). The MLR‐IPCW approach results in a calibration scatter plot, providing extra insight about the calibration. We simulated data with varying levels of censoring and evaluated the ability of each method to estimate the calibration curve for a set of predicted transition probabilities. We also developed and evaluated the calibration of a model predicting the incidence of cardiovascular disease, type 2 diabetes and chronic kidney disease among a cohort of patients derived from linked primary and secondary healthcare records.
Results
The pseudo‐value, BLR‐IPCW, and MLR‐IPCW approaches give unbiased estimates of the calibration curves under random censoring. These methods remained predominately unbiased in the presence of independent censoring, even if the censoring mechanism was strongly associated with the outcome, with bias concentrated in low‐density regions of predicted transition probabilities.
Conclusions
We recommend implementing either the pseudo‐value or BLR‐IPCW approaches to produce a calibration curve, combined with the MLR‐IPCW approach to produce a calibration scatter plot. The methods have been incorporated into the “calibmsm” R package available on CRAN.
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Centre for Health Informatics, Imaging and Data Science, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK.
- NIHR Manchester Biomedical Research Centre, University of Manchester, Manchester, UK.
- Institute of Applied Health Research, University of Birmingham, Birmingham, UK.
- Centre for Epidemiology Versus Arthritis, Centre for Musculoskeletal Research, Manchester Academic Health Science Centre, University of Manchester, Manchester, UK.
- Centre for Biostatistics, Manchester Academic Health Science Centre, University of Manchester, Manchester, UK.
- Keele Cardiovascular Research Group, Keele University, Stoke-on-Trent, UK.
ORCID profiles: Alexander Pate, Matthew Sperrin, Glen P Martin
Funding
- Medical Research Council
Cite
APA
Pate, A., Sperrin, M., Riley, R. D., Peek, N., Van Staa, T., Sergeant, J. C., Mamas, M. A., Lip, G. Y. H., O'Flaherty, M., Barrowman, M., Buchan, I., & Martin, G. P. (2024). Calibration plots for multistate risk prediction models. Statistics in Medicine, 43(14), 2830-2852. https://doi.org/10.1002/sim.10094
Vancouver
Pate A, Sperrin M, Riley RD, Peek N, Van Staa T, Sergeant JC, et al. Calibration plots for multistate risk prediction models. Statistics in Medicine. 2024;43(14):2830-2852. doi:10.1002/sim.10094
BibTeX
@article{Pate2024,
author = {Pate, Alexander and Sperrin, Matthew and Riley, Richard D and Peek, Niels and Van Staa, Tjeerd and Sergeant, Jamie C and Mamas, Mamas A and Lip, Gregory Y H and O'Flaherty, Martin and Barrowman, Michael and Buchan, Iain and Martin, Glen P},
title = {Calibration plots for multistate risk prediction models},
journal = {Statistics in Medicine},
year = {2024},
volume = {43},
number = {14},
pages = {2830--2852},
doi = {10.1002/sim.10094}
}
EUVIMED