Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges
WIREs Data Mining and Knowledge Discovery · 2023-03 · 13 (2)
DOI: 10.1002/widm.1484 ↗At a glance
Journal article in WIREs Data Mining and Knowledge Discovery, published online on 2023-01-16. Written by Bernd Bischl and 11 co-authors. Cited 852 times so far (Crossref); 174 references. The full text is freely available (open access, CC BY 4.0). Funded by: Bundesministerium für Bildung und Forschung, Deutsche Forschungsgemeinschaft.
Abstract (original language)
Most machine learning algorithms are configured by a set of hyperparameters whose values must be carefully chosen and which often considerably impact performance. To avoid a time‐consuming and irreproducible manual process of trial‐and‐error to find well‐performing hyperparameter configurations, various automatic hyperparameter optimization (HPO) methods—for example, based on resampling error estimation for supervised machine learning—can be employed. After introducing HPO from a general perspective, this paper reviews important HPO methods, from simple techniques such as grid or random search to more advanced methods like evolution strategies, Bayesian optimization, Hyperband, and racing. This work gives practical recommendations regarding important choices to be made when conducting HPO, including the HPO algorithms themselves, performance evaluation, how to combine HPO with machine learning pipelines, runtime improvements, and parallelization. This article is categorized under: Algorithmic Development > Statistics Technologies > Machine Learning Technologies > Prediction
Funding
- Bundesministerium für Bildung und Forschung
- Deutsche Forschungsgemeinschaft
Cite
APA
Bischl, B., Binder, M., Lang, M., Pielok, T., Richter, J., Coors, S., Thomas, J., Ullmann, T., Becker, M., Boulesteix, A., Deng, D., & Lindauer, M. (2023). Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges. WIREs Data Mining and Knowledge Discovery, 13(2). https://doi.org/10.1002/widm.1484
Vancouver
Bischl B, Binder M, Lang M, Pielok T, Richter J, Coors S, et al. Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges. WIREs Data Mining and Knowledge Discovery. 2023;13(2). doi:10.1002/widm.1484
BibTeX
@article{Bischl2023,
author = {Bischl, Bernd and Binder, Martin and Lang, Michel and Pielok, Tobias and Richter, Jakob and Coors, Stefan and Thomas, Janek and Ullmann, Theresa and Becker, Marc and Boulesteix, Anne‐Laure and Deng, Difan and Lindauer, Marius},
title = {Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges},
journal = {WIREs Data Mining and Knowledge Discovery},
year = {2023},
volume = {13},
number = {2},
doi = {10.1002/widm.1484}
}
EUVIMED