EUVIMEDOpdag EUVIMED
Open access · CC BY 4.0

Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges

Bernd Bischl, Martin Binder, Michel Lang, Tobias Pielok, Jakob Richter, Stefan Coors, Janek Thomas, Theresa Ullmann, Marc Becker, Anne‐Laure Boulesteix, Difan Deng, Marius Lindauer

WIREs Data Mining and Knowledge Discovery · 2023-03 · 13 (2)

DOI: 10.1002/widm.1484 ↗

At a glance

Journal article in WIREs Data Mining and Knowledge Discovery, published online on 2023-01-16. Written by Bernd Bischl and 11 co-authors. Cited 852 times so far (Crossref); 174 references. The full text is freely available (open access, CC BY 4.0). Funded by: Bundesministerium für Bildung und Forschung, Deutsche Forschungsgemeinschaft.

AbstractBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Most machine learning algorithms are configured by a set of hyperparameters whose values must be carefully chosen and which often considerably impact performance. To avoid a time‐consuming and irreproducible manual process of trial‐and‐error to find well‐performing hyperparameter configurations, various automatic hyperparameter optimization (HPO) methods—for example, based on resampling error estimation for supervised machine learning—can be employed. After introducing HPO from a general perspective, this paper reviews important HPO methods, from simple techniques such as grid or random search to more advanced methods like evolution strategies, Bayesian optimization, Hyperband, and racing. This work gives practical recommendations regarding important choices to be made when conducting HPO, including the HPO algorithms themselves, performance evaluation, how to combine HPO with machine learning pipelines, runtime improvements, and parallelization. This article is categorized under: Algorithmic Development > Statistics Technologies > Machine Learning Technologies > Prediction

Funding

  • Bundesministerium für Bildung und Forschung
  • Deutsche Forschungsgemeinschaft

Cite

APA

Bischl, B., Binder, M., Lang, M., Pielok, T., Richter, J., Coors, S., Thomas, J., Ullmann, T., Becker, M., Boulesteix, A., Deng, D., & Lindauer, M. (2023). Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges. WIREs Data Mining and Knowledge Discovery, 13(2). https://doi.org/10.1002/widm.1484

Vancouver

Bischl B, Binder M, Lang M, Pielok T, Richter J, Coors S, et al. Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges. WIREs Data Mining and Knowledge Discovery. 2023;13(2). doi:10.1002/widm.1484

BibTeX
@article{Bischl2023,
  author = {Bischl, Bernd and Binder, Martin and Lang, Michel and Pielok, Tobias and Richter, Jakob and Coors, Stefan and Thomas, Janek and Ullmann, Theresa and Becker, Marc and Boulesteix, Anne‐Laure and Deng, Difan and Lindauer, Marius},
  title = {Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges},
  journal = {WIREs Data Mining and Knowledge Discovery},
  year = {2023},
  volume = {13},
  number = {2},
  doi = {10.1002/widm.1484}
}
Start for free →