EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Analyzing inexact hypergradients for bilevel learning

Matthias J Ehrhardt, Lindon Roberts

IMA Journal of Applied Mathematics · 21.06.2024 · 89 (1) · 254-278

DOI: 10.1093/imamat/hxad035 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IMA Journal of Applied Mathematics, online erschienen am 30.11.2023. Verfasst von Matthias J Ehrhardt und 1 weiteren Autor:innen. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 36 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: Engineering and Physical Sciences Research Council, Leverhulme Trust.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Estimating hyperparameters has been a long-standing problem in machine learning. We consider the case where the task at hand is modeled as the solution to an optimization problem. Here the exact gradient with respect to the hyperparameters cannot be feasibly computed and approximate strategies are required. We introduce a unified framework for computing hypergradients that generalizes existing methods based on the implicit function theorem and automatic differentiation/backpropagation, showing that these two seemingly disparate approaches are actually tightly connected. Our framework is extremely flexible, allowing its subproblems to be solved with any suitable method, to any degree of accuracy. We derive a priori and computable a posteriori error bounds for all our methods and numerically show that our a posteriori bounds are usually more accurate. Our numerical results also show that, surprisingly, for efficient bilevel optimization, the choice of hypergradient algorithm is at least as important as the choice of lower-level solver.

Förderung

  • Engineering and Physical Sciences Research Council
  • Leverhulme Trust

Zitieren

APA

Ehrhardt, M. J., & Roberts, L. (2024). Analyzing inexact hypergradients for bilevel learning. IMA Journal of Applied Mathematics, 89(1), 254-278. https://doi.org/10.1093/imamat/hxad035

Vancouver

Ehrhardt MJ, Roberts L. Analyzing inexact hypergradients for bilevel learning. IMA Journal of Applied Mathematics. 2024;89(1):254-278. doi:10.1093/imamat/hxad035

BibTeX
@article{Ehrhardt2024,
  author = {Ehrhardt, Matthias J and Roberts, Lindon},
  title = {Analyzing inexact hypergradients for bilevel learning},
  journal = {IMA Journal of Applied Mathematics},
  year = {2024},
  volume = {89},
  number = {1},
  pages = {254--278},
  doi = {10.1093/imamat/hxad035}
}
Kostenlos starten →