EUVIMEDOpdag EUVIMED
Open access · CC BY-NC 4.0

Propensity‐based standardization to enhance the validation and interpretation of prediction model discrimination for a target population

Valentijn M T de Jong iD, Jeroen Hoogland iD, Karel G M Moons, Richard D Riley iD, Tri-Long Nguyen iD, Thomas P A Debray iD

Statistics in Medicine · 2023-08-30 · 42 (19) · 3508-3528

DOI: 10.1002/sim.9817 ↗

At a glance

Journal article in Statistics in Medicine, published online on 2023-06-13. Written by Valentijn M T de Jong and 5 co-authors. Institution: Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands. Major MEDLINE topics: Diagnosis-Related Groups, Benchmarking. Cited 7 times so far (Crossref); 54 references. The full text is freely available (open access, CC BY-NC 4.0). Funded by: European Commission.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

External validation of the discriminative ability of prediction models is of key importance. However, the interpretation of such evaluations is challenging, as the ability to discriminate depends on both the sample characteristics (ie, case‐mix) and the generalizability of predictor coefficients, but most discrimination indices do not provide any insight into their respective contributions. To disentangle differences in discriminative ability across external validation samples due to a lack of model generalizability from differences in sample characteristics, we propose propensity‐weighted measures of discrimination. These weighted metrics, which are derived from propensity scores for sample membership, are standardized for case‐mix differences between the model development and validation samples, allowing for a fair comparison of discriminative ability in terms of model characteristics in a target population of interest. We illustrate our methods with the validation of eight prediction models for deep vein thrombosis in 12 external validation data sets and assess our methods in a simulation study. In the illustrative example, propensity score standardization reduced between‐study heterogeneity of discrimination, indicating that between‐study variability was partially attributable to case‐mix. The simulation study showed that only flexible propensity‐score methods (allowing for non‐linear effects) produced unbiased estimates of model discrimination in the target population, and only when the positivity assumption was met. Propensity score‐based standardization may facilitate the interpretation of (heterogeneity in) discriminative ability of a prediction model as observed across multiple studies, and may guide model updating strategies for a particular target population. Careful propensity score modeling with attention for non‐linear relations is recommended.

Subjects

MeSH terms (MEDLINE)

Diagnosis-Related GroupsBenchmarkingHumansComputer Simulation

Author keywords

concordanceexternal validationprediction modelpropensity scorestandardization

Authors and institutions

  • Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.
  • Data Analytics and Methods Task Force, European Medicines Agency, Amsterdam, The Netherlands.
  • Department of Epidemiology and Data Science, Amsterdam University Medical Centers, Amsterdam, The Netherlands.
  • Institute of Applied Health Research, College of Medical and Dental Sciences, University of Birmingham, Birmingham, UK.
  • Section of Epidemiology, Department of Public Health, Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark.
  • Smart Data Analysis and Statistics, Utrecht, The Netherlands.

ORCID profiles: Valentijn M T de Jong, Jeroen Hoogland, Richard D Riley, Tri-Long Nguyen, Thomas P A Debray

Funding

  • European Commission

Cite

APA

de Jong, V. M. T., Hoogland, J., Moons, K. G. M., Riley, R. D., Nguyen, T. L., & Debray, T. P. A. (2023). Propensity‐based standardization to enhance the validation and interpretation of prediction model discrimination for a target population. Statistics in Medicine, 42(19), 3508-3528. https://doi.org/10.1002/sim.9817

Vancouver

de Jong VMT, Hoogland J, Moons KGM, Riley RD, Nguyen TL, Debray TPA. Propensity‐based standardization to enhance the validation and interpretation of prediction model discrimination for a target population. Statistics in Medicine. 2023;42(19):3508-3528. doi:10.1002/sim.9817

BibTeX
@article{deJong2023,
  author = {de Jong, Valentijn M T and Hoogland, Jeroen and Moons, Karel G M and Riley, Richard D and Nguyen, Tri-Long and Debray, Thomas P A},
  title = {Propensity‐based standardization to enhance the validation and interpretation of prediction model discrimination for a target population},
  journal = {Statistics in Medicine},
  year = {2023},
  volume = {42},
  number = {19},
  pages = {3508--3528},
  doi = {10.1002/sim.9817}
}
Start for free →