What Kinds of Functions Do Deep Neural Networks Learn? Insights from Variational Spline Theory
SIAM Journal on Mathematics of Data Science · 2022-06 · 4 (2) · 464-489
DOI: 10.1137/21m1418642 ↗Kurz gefasst
Fachartikel in der Zeitschrift SIAM Journal on Mathematics of Data Science, online erschienen am 13.04.2022. Verfasst von Rahul Parhi und 1 weiteren Autor:innen. Die Arbeit wurde bisher 33-mal zitiert (Crossref) und nennt 52 Literaturangaben. Gefördert von: Air Force Office of Scientific Research, Office of Naval Research, National Science Foundation.
Abstract (Originalsprache)
We develop a variational framework to understand the properties of functions learned by fitting deep neural networks with rectified linear unit (ReLU) activations to data. We propose a new function space, which is related to classical bounded variation-type spaces, that captures the compositional structure associated with deep neural networks. We derive a representer theorem showing that deep ReLU networks are solutions to regularized data-fitting problems over functions from this space. The function space consists of compositions of functions from the Banach space of second-order bounded variation in the Radon domain. This Banach space has a sparsity-promoting norm, giving insight into the role of sparsity in deep neural networks. The neural network solutions have skip connections and rank-bounded weight matrices, providing new theoretical support for these common architectural choices. The variational problem we study can be recast as a finite-dimensional neural network training problem with regularization schemes related to the notions of weight decay and path-norm regularization. Finally, our analysis builds on techniques from variational spline theory, providing new connections between deep neural networks and splines.
Förderung
- Air Force Office of Scientific Research
- Office of Naval Research
- National Science Foundation
Zitieren
APA
Parhi, R., & Nowak, R. D. (2022). What Kinds of Functions Do Deep Neural Networks Learn? Insights from Variational Spline Theory. SIAM Journal on Mathematics of Data Science, 4(2), 464-489. https://doi.org/10.1137/21m1418642
Vancouver
Parhi R, Nowak RD. What Kinds of Functions Do Deep Neural Networks Learn? Insights from Variational Spline Theory. SIAM Journal on Mathematics of Data Science. 2022;4(2):464-489. doi:10.1137/21m1418642
BibTeX
@article{Parhi2022,
author = {Parhi, Rahul and Nowak, Robert D.},
title = {What Kinds of Functions Do Deep Neural Networks Learn? Insights from Variational Spline Theory},
journal = {SIAM Journal on Mathematics of Data Science},
year = {2022},
volume = {4},
number = {2},
pages = {464--489},
doi = {10.1137/21m1418642}
}
EUVIMED