EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Embedding Principle: A Hierarchical Structure of Loss Landscape of Deep Neural Networks

Yaoyu Zhang, Yuqing Li, Zhongwang Zhang, Tao Luo, Zhi-Qin John Xu

Journal of Machine Learning · 2024

DOI: 10.4208/jml.220108 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Machine Learning, 2024. Verfasst von Yaoyu Zhang und 4 weiteren Autor:innen. Die Arbeit wurde bisher 7-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We prove a general Embedding Principle of loss landscape of deep neural networks (NNs) that unravels a hierarchical structure of the loss landscape of NNs, i.e., loss landscape of an NN contains all critical points of all the narrower NNs. This result is obtained by constructing a class of critical embeddings which map any critical point of a narrower NN to a critical point of the target NN with the same output function. By discovering a wide class of general compatible critical embeddings, we provide a gross estimate of the dimension of critical submanifolds embedded from critical points of narrower NNs. We further prove an irreversibility property of any critical embedding that the number of negative/zero/positive eigenvalues of the Hessian matrix of a critical point may increase but never decrease as an NN becomes wider through the embedding. Using a special realization of general compatible critical embedding, we prove a stringent necessary condition for being a “truly-bad” critical point that never becomes a strict-saddle point through any critical embedding. This result implies the commonplace of strict-saddle points in wide NNs, which may be an important reason underlying the easy optimization of wide NNs widely observed in practice.

Zitieren

APA

Zhang, Y., Li, Y., Zhang, Z., Luo, T., & Xu, Z. J. (2024). Embedding Principle: A Hierarchical Structure of Loss Landscape of Deep Neural Networks. Journal of Machine Learning. https://doi.org/10.4208/jml.220108

Vancouver

Zhang Y, Li Y, Zhang Z, Luo T, Xu ZJ. Embedding Principle: A Hierarchical Structure of Loss Landscape of Deep Neural Networks. Journal of Machine Learning. 2024. doi:10.4208/jml.220108

BibTeX
@article{Zhang2024,
  author = {Zhang, Yaoyu and Li, Yuqing and Zhang, Zhongwang and Luo, Tao and Xu, Zhi-Qin John},
  title = {Embedding Principle: A Hierarchical Structure of Loss Landscape of Deep Neural Networks},
  journal = {Journal of Machine Learning},
  year = {2024},
  doi = {10.4208/jml.220108}
}
Kostenlos starten →