EUVIMEDDescobrir EUVIMED
Open access · CC BY-NC 4.0

A comparison of hyperparameter tuning procedures for clinical prediction models: A simulation study

Zoë S Dunias iD, Ben Van Calster iD, Dirk Timmerman iD, Anne-Laure Boulesteix iD, Maarten van Smeden iD

Statistics in Medicine · 2024-03-15 · 43 (6) · 1119-1134

DOI: 10.1002/sim.9932 ↗

At a glance

Journal article in Statistics in Medicine, published online on 2024-01-08. Written by Zoë S Dunias and 4 co-authors. Institution: Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands. Major MEDLINE topics: Research Design. Cited 40 times so far (Crossref); 36 references. The full text is freely available (open access, CC BY-NC 4.0).

ResumoSubjectsDados bibliográficosCiteRelated studies

Abstract (original language)

Tuning hyperparameters, such as the regularization parameter in Ridge or Lasso regression, is often aimed at improving the predictive performance of risk prediction models. In this study, various hyperparameter tuning procedures for clinical prediction models were systematically compared and evaluated in low‐dimensional data. The focus was on out‐of‐sample predictive performance (discrimination, calibration, and overall prediction error) of risk prediction models developed using Ridge, Lasso, Elastic Net, or Random Forest. The influence of sample size, number of predictors and events fraction on performance of the hyperparameter tuning procedures was studied using extensive simulations. The results indicate important differences between tuning procedures in calibration performance, while generally showing similar discriminative performance. The one‐standard‐error rule for tuning applied to cross‐validation (1SE CV) often resulted in severe miscalibration. Standard non‐repeated and repeated cross‐validation (both 5‐fold and 10‐fold) performed similarly well and outperformed the other tuning procedures. Bootstrap showed a slight tendency to more severe miscalibration than standard cross‐validation‐based tuning procedures. Differences between tuning procedures were larger for smaller sample sizes, lower events fractions and fewer predictors. These results imply that the choice of tuning procedure can have a profound influence on the predictive performance of prediction models. The results support the application of standard 5‐fold or 10‐fold cross‐validation that minimizes out‐of‐sample prediction error. Despite an increased computational burden, we found no clear benefit of repeated over non‐repeated cross‐validation for hyperparameter tuning. We warn against the potentially detrimental effects on model calibration of the popular 1SE CV rule for tuning prediction models in low‐dimensional settings.

Subjects

MeSH terms (MEDLINE)

Research DesignHumansComputer SimulationSample Size

Author keywords

Random Forestcross-validationhyperparameter tuningpenalized regressionprediction models

Authors and institutions

  • Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands.
  • Department of Development and Regeneration, KU Leuven, Leuven, Belgium.
  • Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands.
  • Department of Obstetrics and Gynecology, University Hospitals Leuven, Leuven, Belgium.
  • Institute for Medical Information Processing, Biometry and Epidemiology, University of Munich, Munich, Germany.
  • Munich Center for Machine Learning (MCML), LMU Munich, Munich, Germany.

ORCID profiles: Zoë S Dunias, Ben Van Calster, Dirk Timmerman, Anne-Laure Boulesteix, Maarten van Smeden

Cite

APA

Dunias, Z. S., Van Calster, B., Timmerman, D., Boulesteix, A. L., & van Smeden, M. (2024). A comparison of hyperparameter tuning procedures for clinical prediction models: A simulation study. Statistics in Medicine, 43(6), 1119-1134. https://doi.org/10.1002/sim.9932

Vancouver

Dunias ZS, Van Calster B, Timmerman D, Boulesteix AL, van Smeden M. A comparison of hyperparameter tuning procedures for clinical prediction models: A simulation study. Statistics in Medicine. 2024;43(6):1119-1134. doi:10.1002/sim.9932

BibTeX
@article{Dunias2024,
  author = {Dunias, Zoë S and Van Calster, Ben and Timmerman, Dirk and Boulesteix, Anne-Laure and van Smeden, Maarten},
  title = {A comparison of hyperparameter tuning procedures for clinical prediction models: A simulation study},
  journal = {Statistics in Medicine},
  year = {2024},
  volume = {43},
  number = {6},
  pages = {1119--1134},
  doi = {10.1002/sim.9932}
}
Start for free →