EUVIMEDDescoperă EUVIMED
Open access · CC BY-NC 4.0

Evaluating individualized treatment effect predictions: A model‐based perspective on discrimination and calibration assessment

J Hoogland iD, O Efthimiou iD, T L Nguyen iD, T P A Debray iD

Statistics in Medicine · 2024

DOI: 10.1002/sim.10186 ↗

At a glance

Journal article in Statistics in Medicine, 2024. Written by J Hoogland and 3 co-authors. Institution: Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands. Major MEDLINE topics: Models, Statistical, Randomized Controlled Trials as Topic, Stroke, Computer Simulation. Cited 14 times so far (Crossref); 65 references. The full text is freely available (open access, CC BY-NC 4.0). Funded by: Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung, European Union's Horizon 2020 research and innovation program, ZonMw.

RezumatSubjectsDate bibliograficeCiteRelated studies

Abstract (original language)

In recent years, there has been a growing interest in the prediction of individualized treatment effects. While there is a rapidly growing literature on the development of such models, there is little literature on the evaluation of their performance. In this paper, we aim to facilitate the validation of prediction models for individualized treatment effects. The estimands of interest are defined based on the potential outcomes framework, which facilitates a comparison of existing and novel measures. In particular, we examine existing measures of discrimination for benefit (variations of the c‐for‐benefit), and propose model‐based extensions to the treatment effect setting for discrimination and calibration metrics that have a strong basis in outcome risk prediction. The main focus is on randomized trial data with binary endpoints and on models that provide individualized treatment effect predictions and potential outcome predictions. We use simulated data to provide insight into the characteristics of the examined discrimination and calibration statistics under consideration, and further illustrate all methods in a trial of acute ischemic stroke treatment. The results show that the proposed model‐based statistics had the best characteristics in terms of bias and accuracy. While resampling methods adjusted for the optimism of performance estimates in the development data, they had a high variance across replications that limited their accuracy. Therefore, individualized treatment effect models are best validated in independent data. To aid implementation, a software implementation of the proposed methods was made available in R .

Subjects

MeSH terms (MEDLINE)

Models, StatisticalRandomized Controlled Trials as TopicStrokeComputer SimulationHumansCalibrationTreatment OutcomePrecision Medicine

Author keywords

calibrationdiscriminationindividualizedpredictiontreatment effect

Authors and institutions

  • Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands.
  • Epidemiology and Data Science, Amsterdam University Medical Center, Amsterdam, The Netherlands.
  • Institute of Primary Health Care (BIHAM), University of Bern, Bern, Switzerland.
  • Institute of Social and Preventive Medicine (ISPM), University of Bern, Bern, Switzerland.
  • Section of Epidemiology, Department of Public Health, University of Copenhagen, Copenhagen, Denmark.
  • Smart Data Analysis and Statistics B.V., Utrecht, The Netherlands.

ORCID profiles: J Hoogland, O Efthimiou, T L Nguyen, T P A Debray

Funding

  • Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung
  • European Union's Horizon 2020 research and innovation program
  • ZonMw

Cite

APA

Hoogland, J., Efthimiou, O., Nguyen, T. L., & Debray, T. P. A. (2024). Evaluating individualized treatment effect predictions: A model‐based perspective on discrimination and calibration assessment. Statistics in Medicine. https://doi.org/10.1002/sim.10186

Vancouver

Hoogland J, Efthimiou O, Nguyen TL, Debray TPA. Evaluating individualized treatment effect predictions: A model‐based perspective on discrimination and calibration assessment. Statistics in Medicine. 2024. doi:10.1002/sim.10186

BibTeX
@article{Hoogland2024,
  author = {Hoogland, J and Efthimiou, O and Nguyen, T L and Debray, T P A},
  title = {Evaluating individualized treatment effect predictions: A model‐based perspective on discrimination and calibration assessment},
  journal = {Statistics in Medicine},
  year = {2024},
  doi = {10.1002/sim.10186}
}
Start for free →