EUVIMEDEUVIMED entdecken
Crossref

Function-Space Optimality of Neural Architectures with Multivariate Nonlinearities

Rahul Parhi, Michael Unser

SIAM Journal on Mathematics of Data Science · 31.03.2025 · 7 (1) · 110-135

DOI: 10.1137/23m1620971 ↗

Kurz gefasst

Fachartikel in der Zeitschrift SIAM Journal on Mathematics of Data Science, online erschienen am 09.01.2025. Verfasst von Rahul Parhi und 1 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 72 Literaturangaben. Gefördert von: Swiss National Science Foundation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We investigate the function-space optimality (specifically, the Banach-space optimality) of a large class of shallow neural architectures with multivariate nonlinearities/activation functions. To that end, we construct a new family of Banach spaces defined via a regularization operator, the [Formula: see text]-plane transform, and a sparsity-promoting norm. We prove a representer theorem that states that the solution sets to learning problems posed over these Banach spaces are completely characterized by neural architectures with multivariate nonlinearities. These optimal architectures have skip connections and are tightly connected to orthogonal weight normalization and multi-index models, both of which have received recent interest in the neural network community. Our framework is compatible with a number of classical nonlinearities including the rectified linear unit activation function, the norm activation function, and the radial basis functions found in the theory of thin-plate/polyharmonic splines. We also show that the underlying spaces are special instances of reproducing kernel Banach spaces and variation spaces. Our results shed light on the regularity of functions learned by neural networks trained on data, particularly with multivariate nonlinearities, and provide new theoretical motivation for several architectural choices found in practice.

Förderung

  • Swiss National Science Foundation

Zitieren

APA

Parhi, R., & Unser, M. (2025). Function-Space Optimality of Neural Architectures with Multivariate Nonlinearities. SIAM Journal on Mathematics of Data Science, 7(1), 110-135. https://doi.org/10.1137/23m1620971

Vancouver

Parhi R, Unser M. Function-Space Optimality of Neural Architectures with Multivariate Nonlinearities. SIAM Journal on Mathematics of Data Science. 2025;7(1):110-135. doi:10.1137/23m1620971

BibTeX
@article{Parhi2025,
  author = {Parhi, Rahul and Unser, Michael},
  title = {Function-Space Optimality of Neural Architectures with Multivariate Nonlinearities},
  journal = {SIAM Journal on Mathematics of Data Science},
  year = {2025},
  volume = {7},
  number = {1},
  pages = {110--135},
  doi = {10.1137/23m1620971}
}
Kostenlos starten →