On the Convergence, Implicit Bias, and Edge of Stability of Gradient Descent in Deep Learning: Reviewing recent progress [Special Issue on the Mathematics of Deep Learning]
IEEE Signal Processing Magazine · 2026-05 · 43 (3) · 37-51
DOI: 10.1109/msp.2026.3665010 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Signal Processing Magazine, 2026-05. Verfasst von Hancheng Min und 2 weiteren Autor:innen. Die Arbeit nennt 45 Literaturangaben. Gefördert von: NSF, Simons Foundation, ONR MURI.
Abstract (Originalsprache)
Deep neural networks (DNNs) trained via gradient descent (GD) with random initialization and without any regularization enjoy good generalization performance in practice despite being highly overparametrized. To theoretically understand this puzzling phenomenon, many works on convergence analysis for GD algorithms on NNs have been developed over the last half-decade. In this article, we review these research efforts and discuss how they address three specific questions related to this puzzle. The first question is why GD finds a global minimum efficiently, which the literature has addressed by studying what level of overparametrization (width, depth, etc.) and what type of initialization lead to a benign optimization landscape along the training trajectory, facilitating a linear convergence rate of GD. The next question is why the global minimum found by GD generalizes well, which has been addressed by showing that overparametrization induces an implicit simplicity bias along the GD trajectory. More recently, it has been observed that in practice, training DNNs with far larger learning rates than theoretically permissible results in faster convergence and better generalization. This leads to the third question of why faster convergence and better generalization can be achieved with a large learning rate, which has been recently addressed by identifying a self-stabilization mechanism and implicit bias toward flat minima.
Förderung
- NSF
- Simons Foundation
- ONR MURI
Zitieren
APA
Min, H., MacDonald, L. E., & Vidal, R. (2026). On the Convergence, Implicit Bias, and Edge of Stability of Gradient Descent in Deep Learning: Reviewing recent progress [Special Issue on the Mathematics of Deep Learning]. IEEE Signal Processing Magazine, 43(3), 37-51. https://doi.org/10.1109/msp.2026.3665010
Vancouver
Min H, MacDonald LE, Vidal R. On the Convergence, Implicit Bias, and Edge of Stability of Gradient Descent in Deep Learning: Reviewing recent progress [Special Issue on the Mathematics of Deep Learning]. IEEE Signal Processing Magazine. 2026;43(3):37-51. doi:10.1109/msp.2026.3665010
BibTeX
@article{Min2026,
author = {Min, Hancheng and MacDonald, Lachlan Ewen and Vidal, René},
title = {On the Convergence, Implicit Bias, and Edge of Stability of Gradient Descent in Deep Learning: Reviewing recent progress [Special Issue on the Mathematics of Deep Learning]},
journal = {IEEE Signal Processing Magazine},
year = {2026},
volume = {43},
number = {3},
pages = {37--51},
doi = {10.1109/msp.2026.3665010}
}
EUVIMED