EUVIMEDOpdag EUVIMED
Open access

Validation of predicted individual treatment effects in out of sample respondents

Alena Kuhlemeier iD, Thomas Jaki iD, Katie Witkiewitz, Elizabeth A Stuart, M Lee Van Horn

Statistics in Medicine · 2024

DOI: 10.1002/sim.10187 ↗

At a glance

Journal article in Statistics in Medicine, 2024. Written by Alena Kuhlemeier and 4 co-authors. Institution: Center on Alcohol, Substance Use, and Addictions, University of New Mexico, Albuquerque, New Mexico, USA. Major MEDLINE topics: Precision Medicine, Algorithms. Cited 4 times so far (Crossref); 28 references. The full text is freely available in PubMed Central. Funded by: NIAAA NIH HHS, National Institute on Alcohol Abuse and Alcoholism.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Personalized medicine promises the ability to improve patient outcomes by tailoring treatment recommendations to the likelihood that any given patient will respond well to a given treatment. It is important that predictions of treatment response be validated and replicated in independent data to support their use in clinical practice. In this paper, we propose and test an approach for validating predictions of individual treatment effects with continuous outcomes across samples that uses matching in a test (validation) sample to match individuals in the treatment and control arms based on their predicted treatment response and their predicted response under control. To examine the proposed validation approach, we conducted simulations where test data is generated from either an identical, similar, or unrelated process to the training data. We also examined the impact of nuisance variables. To demonstrate the use of this validation procedure in the context of predicting individual treatment effects in the treatment of alcohol use disorder, we apply our validation procedure using data from a clinical trial of combined behavioral and pharmacotherapy treatments. We find that the validation algorithm accurately confirms validation and lack of validation, and also provides insights into cases where test data were generated under similar, but not identical conditions. We also show that the presence of nuisance variables detrimentally impacts algorithm performance, which can be partially reduced though the use of variable selection methods. An advantage of the approach is that it can be widely applied to different predictive methods.

Subjects

MeSH terms (MEDLINE)

Precision MedicineAlgorithmsHumansAlcoholismComputer SimulationTreatment OutcomeModels, StatisticalReproducibility of Results

Author keywords

individual treatment effectspersonalized medicinevalidation

Authors and institutions

  • Center on Alcohol, Substance Use, and Addictions, University of New Mexico, Albuquerque, New Mexico, USA.
  • Chair for Computational Statistics, University of Regensburg, Regensburg, Germany.
  • MRC Biostatistics Unit, University of Cambridge, Cambridge, UK.
  • Department of Biostatistics, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, USA.
  • Department of Individual, Family, & Community Education, College of Education and Human Sciences, University of New Mexico, Albuquerque, New Mexico, USA.

ORCID profiles: Alena Kuhlemeier, Thomas Jaki

Funding

  • NIAAA NIH HHS
  • National Institute on Alcohol Abuse and Alcoholism

Cite

APA

Kuhlemeier, A., Jaki, T., Witkiewitz, K., Stuart, E. A., & Van Horn, M. L. (2024). Validation of predicted individual treatment effects in out of sample respondents. Statistics in Medicine. https://doi.org/10.1002/sim.10187

Vancouver

Kuhlemeier A, Jaki T, Witkiewitz K, Stuart EA, Van Horn ML. Validation of predicted individual treatment effects in out of sample respondents. Statistics in Medicine. 2024. doi:10.1002/sim.10187

BibTeX
@article{Kuhlemeier2024,
  author = {Kuhlemeier, Alena and Jaki, Thomas and Witkiewitz, Katie and Stuart, Elizabeth A and Van Horn, M Lee},
  title = {Validation of predicted individual treatment effects in out of sample respondents},
  journal = {Statistics in Medicine},
  year = {2024},
  doi = {10.1002/sim.10187}
}
Start for free →