A comparison of hyperparameter tuning procedures for clinical prediction models: A simulation study
Statistics in Medicine · 15.03.2024 · 43 (6) · 1119-1134
DOI: 10.1002/sim.9932 ↗Kurz gefasst
Fachartikel in der Zeitschrift Statistics in Medicine, online erschienen am 08.01.2024. Verfasst von Zoë S Dunias und 4 weiteren Autor:innen. Beteiligte Einrichtung: Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands. Hauptthemen laut MEDLINE: Research Design. Die Arbeit wurde bisher 40-mal zitiert (Crossref) und nennt 36 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
Tuning hyperparameters, such as the regularization parameter in Ridge or Lasso regression, is often aimed at improving the predictive performance of risk prediction models. In this study, various hyperparameter tuning procedures for clinical prediction models were systematically compared and evaluated in low‐dimensional data. The focus was on out‐of‐sample predictive performance (discrimination, calibration, and overall prediction error) of risk prediction models developed using Ridge, Lasso, Elastic Net, or Random Forest. The influence of sample size, number of predictors and events fraction on performance of the hyperparameter tuning procedures was studied using extensive simulations. The results indicate important differences between tuning procedures in calibration performance, while generally showing similar discriminative performance. The one‐standard‐error rule for tuning applied to cross‐validation (1SE CV) often resulted in severe miscalibration. Standard non‐repeated and repeated cross‐validation (both 5‐fold and 10‐fold) performed similarly well and outperformed the other tuning procedures. Bootstrap showed a slight tendency to more severe miscalibration than standard cross‐validation‐based tuning procedures. Differences between tuning procedures were larger for smaller sample sizes, lower events fractions and fewer predictors. These results imply that the choice of tuning procedure can have a profound influence on the predictive performance of prediction models. The results support the application of standard 5‐fold or 10‐fold cross‐validation that minimizes out‐of‐sample prediction error. Despite an increased computational burden, we found no clear benefit of repeated over non‐repeated cross‐validation for hyperparameter tuning. We warn against the potentially detrimental effects on model calibration of the popular 1SE CV rule for tuning prediction models in low‐dimensional settings.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands.
- Department of Development and Regeneration, KU Leuven, Leuven, Belgium.
- Department of Biomedical Data Sciences, Leiden University Medical Center, Leiden, The Netherlands.
- Department of Obstetrics and Gynecology, University Hospitals Leuven, Leuven, Belgium.
- Institute for Medical Information Processing, Biometry and Epidemiology, University of Munich, Munich, Germany.
- Munich Center for Machine Learning (MCML), LMU Munich, Munich, Germany.
ORCID-Profile: Zoë S Dunias, Ben Van Calster, Dirk Timmerman, Anne-Laure Boulesteix, Maarten van Smeden
Zitieren
APA
Dunias, Z. S., Van Calster, B., Timmerman, D., Boulesteix, A. L., & van Smeden, M. (2024). A comparison of hyperparameter tuning procedures for clinical prediction models: A simulation study. Statistics in Medicine, 43(6), 1119-1134. https://doi.org/10.1002/sim.9932
Vancouver
Dunias ZS, Van Calster B, Timmerman D, Boulesteix AL, van Smeden M. A comparison of hyperparameter tuning procedures for clinical prediction models: A simulation study. Statistics in Medicine. 2024;43(6):1119-1134. doi:10.1002/sim.9932
BibTeX
@article{Dunias2024,
author = {Dunias, Zoë S and Van Calster, Ben and Timmerman, Dirk and Boulesteix, Anne-Laure and van Smeden, Maarten},
title = {A comparison of hyperparameter tuning procedures for clinical prediction models: A simulation study},
journal = {Statistics in Medicine},
year = {2024},
volume = {43},
number = {6},
pages = {1119--1134},
doi = {10.1002/sim.9932}
}
EUVIMED