Architecture induces structural invariant manifolds of neural network training dynamics
Mathematical Models and Methods in Applied Sciences · 2026-07 · 36 (08) · 1831-1875
DOI: 10.1142/s0218202526420078 ↗Kurz gefasst
Fachartikel in der Zeitschrift Mathematical Models and Methods in Applied Sciences, online erschienen am 17.04.2026. Verfasst von Jiajie Zhao und 2 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 37 Literaturangaben. Gefördert von: National Key R&D Program of China, Natural Science Foundation of China, Natural Science Foundation of Shanghai.
Abstract (Originalsprache)
While architecture is recognized as key to the performance of deep neural networks, its precise effect on training dynamics has been unclear due to the confounding influence of data and loss functions. This paper proposed an analytic framework based on the geometric control theory to characterize the dynamical properties intrinsic to a model’s parameterization. We prove that the Structural Invariant Manifolds (SIMs) of an analytic model [Formula: see text] — submanifolds that confine gradient flow trajectories independent of data and loss — are unions of orbits of the vector field family [Formula: see text]. We then prove that a model’s symmetry, e.g. permutation symmetry for neural networks, induces SIMs. By applying this, we characterize the hierarchy of symmetry-induced SIMs in fully-connected networks, where dynamics exhibit neuron condensation and equivalence to reduced-width networks. For two-layer networks, we prove all SIMs are symmetry-induced, closing the gap between known symmetries and all possible invariants. Overall, by establishing the framework for analyzing SIMs induced by architecture, our work paves the way for a deeper analysis of neural network training dynamics and generalization in the near future.
Förderung
- National Key R&D Program of China
- Natural Science Foundation of China
- Natural Science Foundation of Shanghai
- Shanghai Institute for Mathematics and Interdisciplinary Sciences
Zitieren
APA
Zhao, J., Luo, T., & Zhang, Y. (2026). Architecture induces structural invariant manifolds of neural network training dynamics. Mathematical Models and Methods in Applied Sciences, 36(08), 1831-1875. https://doi.org/10.1142/s0218202526420078
Vancouver
Zhao J, Luo T, Zhang Y. Architecture induces structural invariant manifolds of neural network training dynamics. Mathematical Models and Methods in Applied Sciences. 2026;36(08):1831-1875. doi:10.1142/s0218202526420078
BibTeX
@article{Zhao2026,
author = {Zhao, Jiajie and Luo, Tao and Zhang, Yaoyu},
title = {Architecture induces structural invariant manifolds of neural network training dynamics},
journal = {Mathematical Models and Methods in Applied Sciences},
year = {2026},
volume = {36},
number = {08},
pages = {1831--1875},
doi = {10.1142/s0218202526420078}
}
EUVIMED