EUVIMEDEUVIMED entdecken
Crossref

Gradient Flow Equations for Deep Linear Neural Networks: A Survey from a Network Perspective

Joel Wendin, Claudio Altafini

SIAM Review · 30.06.2026 · 68 (2) · 293-345

DOI: 10.1137/24m1715519 ↗

Kurz gefasst

Fachartikel in der Zeitschrift SIAM Review, online erschienen am 13.05.2026. Verfasst von Joel Wendin und 1 weiteren Autor:innen. Die Arbeit nennt 45 Literaturangaben. Gefördert von: Excellence Center at Linköping – Lund in Information Technology, Svenska Forskningsrådet Formas.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This paper surveys recent progress in understanding the dynamics and loss landscape of the gradient flow equations associated with deep linear neural networks, i.e., the gradient descent training dynamics (in the limit when the step size goes to 0) of deep neural networks missing the activation functions and subject to quadratic loss functions. When formulated in terms of the adjacency matrix of the neural network, as is done in this paper, these gradient flow equations form a class of converging matrix ODEs which is nilpotent, polynomial, isospectral, and with conservation laws. A detailed description of the loss landscape shows that it is described in detail and is characterized by infinitely-many global minima and saddle points, both strict and nonstrict, but that it lacks local minima and maxima. The loss function itself is a positive semidefinite Lyapunov function for the gradient flow, and its level sets are unbounded invariant sets of critical points with critical values that correspond to the amount of singular values of the input-output data learnt by the gradient along a certain trajectory. The adjacency matrix representation we use in the paper allows us to highlight the existence of a quotient space structure in which each critical value of the loss function is represented only once, while all other critical points with the same critical value belong to the fiber associated to the quotient space. It also allows us to easily determine stable and unstable submanifolds at the saddle points, even when the Hessian fails to obtain them.

Förderung

  • Excellence Center at Linköping – Lund in Information Technology
  • Svenska Forskningsrådet Formas

Zitieren

APA

Wendin, J., & Altafini, C. (2026). Gradient Flow Equations for Deep Linear Neural Networks: A Survey from a Network Perspective. SIAM Review, 68(2), 293-345. https://doi.org/10.1137/24m1715519

Vancouver

Wendin J, Altafini C. Gradient Flow Equations for Deep Linear Neural Networks: A Survey from a Network Perspective. SIAM Review. 2026;68(2):293-345. doi:10.1137/24m1715519

BibTeX
@article{Wendin2026,
  author = {Wendin, Joel and Altafini, Claudio},
  title = {Gradient Flow Equations for Deep Linear Neural Networks: A Survey from a Network Perspective},
  journal = {SIAM Review},
  year = {2026},
  volume = {68},
  number = {2},
  pages = {293--345},
  doi = {10.1137/24m1715519}
}
Kostenlos starten →