EUVIMEDEUVIMED entdecken
Crossref

A Systematic Study on Early Stopping Metrics in HPO and the Implications of Uncertainty

Jiawei Guan, Feng Zhang, Jiesong Liu, Xiaoyong Du, Xipeng Shen

Proceedings of the VLDB Endowment · 2025-02 · 18 (6) · 1551-1564

DOI: 10.14778/3725688.3725689 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the VLDB Endowment, online erschienen am 29.08.2025. Verfasst von Jiawei Guan und 4 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 60 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The development of hyperparameter optimization (HPO) algorithms is an important topic within both the machine learning and data management domains. While numerous strategies employing early stopping mechanisms have been proposed to bolster HPO efficiency, there remains a notable deficiency in understanding how the selection of early stopping metrics influences the reliability of early stopping decisions and, by extension, the broader HPO outcomes. This paper undertakes a systematic exploration of the impact of metric selection on the effectiveness of early stopping-based HPO. Specifically, we introduce a set of metrics that incorporate uncertainty and highlight their practical significance in enhancing the reliability of early stopping decisions. Our empirical experiments on HPO and NAS benchmarks show that using training loss as an early stopping metric in the early training stages improves HPO outcomes by up to 24.76% compared to the more widely accepted validation loss. Furthermore, integrating uncertainty into the metric yields an additional improvement of up to 4% under budget constraints, translating into meaningful resource savings and scalability benefits in large-scale HPO scenarios. These findings demonstrate the critical role of metric selection while shedding light on the potential implications of integrating uncertainty as a metric. This research provides empirical insights that serve as a compass for the selection and formulation of metrics, thereby contributing to a more profound comprehension of mechanisms underpinning early stopping-based HPO.

Zitieren

APA

Guan, J., Zhang, F., Liu, J., Du, X., & Shen, X. (2025). A Systematic Study on Early Stopping Metrics in HPO and the Implications of Uncertainty. Proceedings of the VLDB Endowment, 18(6), 1551-1564. https://doi.org/10.14778/3725688.3725689

Vancouver

Guan J, Zhang F, Liu J, Du X, Shen X. A Systematic Study on Early Stopping Metrics in HPO and the Implications of Uncertainty. Proceedings of the VLDB Endowment. 2025;18(6):1551-1564. doi:10.14778/3725688.3725689

BibTeX
@article{Guan2025,
  author = {Guan, Jiawei and Zhang, Feng and Liu, Jiesong and Du, Xiaoyong and Shen, Xipeng},
  title = {A Systematic Study on Early Stopping Metrics in HPO and the Implications of Uncertainty},
  journal = {Proceedings of the VLDB Endowment},
  year = {2025},
  volume = {18},
  number = {6},
  pages = {1551--1564},
  doi = {10.14778/3725688.3725689}
}
Kostenlos starten →