EUVIMEDEUVIMED entdecken
Crossref

Architecture induces structural invariant manifolds of neural network training dynamics

Jiajie Zhao, Tao Luo, Yaoyu Zhang

Mathematical Models and Methods in Applied Sciences · 2026-07 · 36 (08) · 1831-1875

DOI: 10.1142/s0218202526420078 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Mathematical Models and Methods in Applied Sciences, online erschienen am 17.04.2026. Verfasst von Jiajie Zhao und 2 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 37 Literaturangaben. Gefördert von: National Key R&D Program of China, Natural Science Foundation of China, Natural Science Foundation of Shanghai.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

While architecture is recognized as key to the performance of deep neural networks, its precise effect on training dynamics has been unclear due to the confounding influence of data and loss functions. This paper proposed an analytic framework based on the geometric control theory to characterize the dynamical properties intrinsic to a model’s parameterization. We prove that the Structural Invariant Manifolds (SIMs) of an analytic model [Formula: see text] — submanifolds that confine gradient flow trajectories independent of data and loss — are unions of orbits of the vector field family [Formula: see text]. We then prove that a model’s symmetry, e.g. permutation symmetry for neural networks, induces SIMs. By applying this, we characterize the hierarchy of symmetry-induced SIMs in fully-connected networks, where dynamics exhibit neuron condensation and equivalence to reduced-width networks. For two-layer networks, we prove all SIMs are symmetry-induced, closing the gap between known symmetries and all possible invariants. Overall, by establishing the framework for analyzing SIMs induced by architecture, our work paves the way for a deeper analysis of neural network training dynamics and generalization in the near future.

Förderung

  • National Key R&D Program of China
  • Natural Science Foundation of China
  • Natural Science Foundation of Shanghai
  • Shanghai Institute for Mathematics and Interdisciplinary Sciences

Zitieren

APA

Zhao, J., Luo, T., & Zhang, Y. (2026). Architecture induces structural invariant manifolds of neural network training dynamics. Mathematical Models and Methods in Applied Sciences, 36(08), 1831-1875. https://doi.org/10.1142/s0218202526420078

Vancouver

Zhao J, Luo T, Zhang Y. Architecture induces structural invariant manifolds of neural network training dynamics. Mathematical Models and Methods in Applied Sciences. 2026;36(08):1831-1875. doi:10.1142/s0218202526420078

BibTeX
@article{Zhao2026,
  author = {Zhao, Jiajie and Luo, Tao and Zhang, Yaoyu},
  title = {Architecture induces structural invariant manifolds of neural network training dynamics},
  journal = {Mathematical Models and Methods in Applied Sciences},
  year = {2026},
  volume = {36},
  number = {08},
  pages = {1831--1875},
  doi = {10.1142/s0218202526420078}
}
Kostenlos starten →