EUVIMEDEUVIMED entdecken
Crossref

What Kinds of Functions Do Deep Neural Networks Learn? Insights from Variational Spline Theory

Rahul Parhi, Robert D. Nowak

SIAM Journal on Mathematics of Data Science · 2022-06 · 4 (2) · 464-489

DOI: 10.1137/21m1418642 ↗

Kurz gefasst

Fachartikel in der Zeitschrift SIAM Journal on Mathematics of Data Science, online erschienen am 13.04.2022. Verfasst von Rahul Parhi und 1 weiteren Autor:innen. Die Arbeit wurde bisher 33-mal zitiert (Crossref) und nennt 52 Literaturangaben. Gefördert von: Air Force Office of Scientific Research, Office of Naval Research, National Science Foundation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We develop a variational framework to understand the properties of functions learned by fitting deep neural networks with rectified linear unit (ReLU) activations to data. We propose a new function space, which is related to classical bounded variation-type spaces, that captures the compositional structure associated with deep neural networks. We derive a representer theorem showing that deep ReLU networks are solutions to regularized data-fitting problems over functions from this space. The function space consists of compositions of functions from the Banach space of second-order bounded variation in the Radon domain. This Banach space has a sparsity-promoting norm, giving insight into the role of sparsity in deep neural networks. The neural network solutions have skip connections and rank-bounded weight matrices, providing new theoretical support for these common architectural choices. The variational problem we study can be recast as a finite-dimensional neural network training problem with regularization schemes related to the notions of weight decay and path-norm regularization. Finally, our analysis builds on techniques from variational spline theory, providing new connections between deep neural networks and splines.

Förderung

  • Air Force Office of Scientific Research
  • Office of Naval Research
  • National Science Foundation

Zitieren

APA

Parhi, R., & Nowak, R. D. (2022). What Kinds of Functions Do Deep Neural Networks Learn? Insights from Variational Spline Theory. SIAM Journal on Mathematics of Data Science, 4(2), 464-489. https://doi.org/10.1137/21m1418642

Vancouver

Parhi R, Nowak RD. What Kinds of Functions Do Deep Neural Networks Learn? Insights from Variational Spline Theory. SIAM Journal on Mathematics of Data Science. 2022;4(2):464-489. doi:10.1137/21m1418642

BibTeX
@article{Parhi2022,
  author = {Parhi, Rahul and Nowak, Robert D.},
  title = {What Kinds of Functions Do Deep Neural Networks Learn? Insights from Variational Spline Theory},
  journal = {SIAM Journal on Mathematics of Data Science},
  year = {2022},
  volume = {4},
  number = {2},
  pages = {464--489},
  doi = {10.1137/21m1418642}
}
Kostenlos starten →