Efficient Gradient-Based Methods for Bilevel Learning via Recycling Krylov Subspaces
SIAM Journal on Scientific Computing · 30.06.2026 · 48 (3) · A1811-A1838
DOI: 10.1137/24m1716884 ↗Kurz gefasst
Fachartikel in der Zeitschrift SIAM Journal on Scientific Computing, online erschienen am 24.06.2026. Verfasst von M. J. Ehrhardt und 2 weiteren Autor:innen. Die Arbeit nennt 78 Literaturangaben. Gefördert von: Engineering and Physical Sciences Research Council Centre for Doctoral Training in Statistical Applied Mathematics at Bath, Bundesministerium für Bildung und Forschung, H2020 Marie Skłodowska-Curie Actions.
Abstract (Originalsprache)
Many optimization problems require hyperparameters, i.e., parameters that must be prespecified in advance of their solution, such as regularization parameters and parametric regularizers in variational regularization methods for inverse problems and dictionaries in compressed sensing. A data-driven approach to determine appropriate hyperparameter values is via a nested optimization framework known as bilevel learning. Even when it is possible to employ a gradient-based solver to the bilevel optimization problem, construction of the gradients, known as hypergradients, is computationally challenging, each one requiring both a solution of a minimization problem and a linear system solve. These systems do not change much during the iterations, which motivates us to apply recycling Krylov subspace methods, wherein information from one linear system solve is reused to solve the next linear system. Existing recycling strategies often employ eigenvector approximations called Ritz vectors. In this work, we propose a novel recycling strategy based on a new concept, Ritz generalized singular vectors, which acknowledge the bilevel setting. Additionally, while existing iterative methods primarily terminate according to the residual norm, the newly introduced Ritz generalized singular vectors allow us to define a new stopping criterion that directly approximates the error of the associated hypergradient. The proposed approach is validated through extensive numerical testing in the context of inverse problems in imaging. Reproducibility of computational results. This paper has been awarded the “SIAM Reproducibility Badge: Code and data available” as a recognition that the authors have followed reproducibility principles valued by SISC and the scientific computing community. Code and data that allow readers to reproduce the results in this paper are available at https://github.com/s-j-scott/bilevel-recycling and in the supplementary materials ( M171688_Supplementary_Materials_1.pdf [177KB], bilevel-recycling-main.zip [80.4MB]). [Formula: see text]
Förderung
- Engineering and Physical Sciences Research Council Centre for Doctoral Training in Statistical Applied Mathematics at Bath
- Bundesministerium für Bildung und Forschung
- H2020 Marie Skłodowska-Curie Actions
- Engineering and Physical Sciences Research Council
Zitieren
APA
Ehrhardt, M. J., Gazzola, S., & Scott, S. J. (2026). Efficient Gradient-Based Methods for Bilevel Learning via Recycling Krylov Subspaces. SIAM Journal on Scientific Computing, 48(3), A1811-A1838. https://doi.org/10.1137/24m1716884
Vancouver
Ehrhardt MJ, Gazzola S, Scott SJ. Efficient Gradient-Based Methods for Bilevel Learning via Recycling Krylov Subspaces. SIAM Journal on Scientific Computing. 2026;48(3):A1811-A1838. doi:10.1137/24m1716884
BibTeX
@article{Ehrhardt2026,
author = {Ehrhardt, M. J. and Gazzola, Silvia and Scott, Sebastian J.},
title = {Efficient Gradient-Based Methods for Bilevel Learning via Recycling Krylov Subspaces},
journal = {SIAM Journal on Scientific Computing},
year = {2026},
volume = {48},
number = {3},
pages = {A1811--A1838},
doi = {10.1137/24m1716884}
}
EUVIMED