ReLU Neural Networks with Linear Layers Are Biased towards Single- and Multi-index Models
SIAM Journal on Mathematics of Data Science · 30.09.2025 · 7 (3) · 1021-1052
DOI: 10.1137/24m1672158 ↗Kurz gefasst
Fachartikel in der Zeitschrift SIAM Journal on Mathematics of Data Science, online erschienen am 09.07.2025. Verfasst von Suzanna Parkinson und 2 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 85 Literaturangaben. Gefördert von: Air Force Office of Scientific Research, National Science Foundation.
Abstract (Originalsprache)
Neural networks often operate in the overparameterized regime, in which there are far more parameters than training samples, allowing the training data to fit perfectly. That is, training the network effectively learns an interpolating function, and properties of the interpolant affect predictions the network will make on new samples. This manuscript explores how properties of such functions learned by neural networks of depth greater than two layers. Our framework considers a family of networks of varying depths that all have the same capacity but different representation costs. The representation cost of a function induced by a neural network architecture is the minimum sum of squared weights needed for the network to represent the function; it reflects the function space bias associated with the architecture. Our results show that adding additional linear layers to the input side of a shallow ReLU network yields a representation cost favoring functions with low mixed variation–-that is, it has limited variation in directions orthogonal to a low-dimensional subspace and can be well approximated by a single- or multi-index model. This bias occurs because minimizing the sum of squared weights of the linear layers is equivalent to minimizing a low-rank promoting Schatten quasi-norm of a single “virtual” weight matrix. Our experiments confirm this behavior in standard network training regimes. They additionally show that linear layers can improve generalization and the learned network is well-aligned with the true latent low-dimensional linear subspace when data is generated using a multi-index model.
Förderung
- Air Force Office of Scientific Research
- National Science Foundation
Zitieren
APA
Parkinson, S., Ongie, G., & Willett, R. (2025). ReLU Neural Networks with Linear Layers Are Biased towards Single- and Multi-index Models. SIAM Journal on Mathematics of Data Science, 7(3), 1021-1052. https://doi.org/10.1137/24m1672158
Vancouver
Parkinson S, Ongie G, Willett R. ReLU Neural Networks with Linear Layers Are Biased towards Single- and Multi-index Models. SIAM Journal on Mathematics of Data Science. 2025;7(3):1021-1052. doi:10.1137/24m1672158
BibTeX
@article{Parkinson2025,
author = {Parkinson, Suzanna and Ongie, Greg and Willett, Rebecca},
title = {ReLU Neural Networks with Linear Layers Are Biased towards Single- and Multi-index Models},
journal = {SIAM Journal on Mathematics of Data Science},
year = {2025},
volume = {7},
number = {3},
pages = {1021--1052},
doi = {10.1137/24m1672158}
}
EUVIMED