EUVIMEDEUVIMED entdecken
Crossref

Global Minima of Overparameterized Neural Networks

Yaim Cooper

SIAM Journal on Mathematics of Data Science · 2021-01 · 3 (2) · 676-691

DOI: 10.1137/19m1308943 ↗

Kurz gefasst

Fachartikel in der Zeitschrift SIAM Journal on Mathematics of Data Science, online erschienen am 13.05.2021. Verfasst von Yaim Cooper. Die Arbeit wurde bisher 22-mal zitiert (Crossref) und nennt 25 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We explore some mathematical features of the loss landscape of overparameterized neural networks. A priori, one might imagine that the loss function looks like a typical function from $\mathbb{R}^d$ to $\mathbb{R}$, in particular, that it has discrete global minima. In this paper, we prove that in at least one important way, the loss function of an overparameterized neural network does not look like a typical function. If a neural net has $d$ parameters and is trained on $n$ data points $(x_i, y_i) \in \mathbb{R}^s \times \mathbb{R}^r$, with $d>r n$, we show that the locus $M$ of global minima of $L$ is usually not discrete but rather an $(d- rn)$-dimensional submanifold of $\mathbb{R}^d$. In practice, neural nets commonly have orders of magnitude more parameters than data points, so this observation implies that $M$ is typically a very-high-dimensional submanifold of $\mathbb{R}^d$.

Zitieren

APA

Cooper, Y. (2021). Global Minima of Overparameterized Neural Networks. SIAM Journal on Mathematics of Data Science, 3(2), 676-691. https://doi.org/10.1137/19m1308943

Vancouver

Cooper Y. Global Minima of Overparameterized Neural Networks. SIAM Journal on Mathematics of Data Science. 2021;3(2):676-691. doi:10.1137/19m1308943

BibTeX
@article{Cooper2021,
  author = {Cooper, Yaim},
  title = {Global Minima of Overparameterized Neural Networks},
  journal = {SIAM Journal on Mathematics of Data Science},
  year = {2021},
  volume = {3},
  number = {2},
  pages = {676--691},
  doi = {10.1137/19m1308943}
}
Kostenlos starten →