Dimension free ridge regression
The Annals of Statistics · 01.12.2024 · 52 (6) · 2879-2912
DOI: 10.1214/24-aos2449 ↗Kurz gefasst
Fachartikel in der Zeitschrift The Annals of Statistics, erschienen am 01.12.2024. Verfasst von Chen Cheng und 1 weiteren Autor:innen. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 41 Literaturangaben.
Abstract (Originalsprache)
Random matrix theory has become a widely useful tool in high-dimensional statistics and theoretical machine learning. However, random matrix theory is largely focused on the proportional asymptotics in which the number of columns grows proportionally to the number of rows of the data matrix. This is not always the most natural setting in statistics where columns correspond to covariates and rows to samples. With the objective to move beyond the proportional asymptotics, we revisit ridge regression (ℓ2-penalized least squares) on i.i.d. data (xi,yi), i≤n, where xi is a feature vector and yi=⟨β,xi⟩+εi∈R is a response. We allow the feature vector to be high-dimensional, or even infinite-dimensional, in which case it belongs to a separable Hilbert space, and assume either zi:=Σ−1/2xi to have i.i.d. entries, or to satisfy a certain convex concentration property. Within this setting, we establish nonasymptotic bounds that approximate the bias and variance of ridge regression in terms of the bias and variance of an “equivalent” sequence model (a regression model with diagonal design matrix). The approximation is up to multiplicative factors bounded by (1±Δ) for some explicitly small Δ. Previously, such an approximation result was known only in the proportional regime and only up to additive errors: in particular, it did not allow to characterize the behavior of the excess risk when this converges to 0. Our general theory recovers earlier results in the proportional regime (with better error rates). As a new application, we obtain a completely explicit and sharp characterization of ridge regression for Hilbert covariates with regularly varying spectrum. Finally, we analyze the overparametrized near-interpolation setting and obtain sharp “benign overfitting” guarantees.
Zitieren
APA
Cheng, C., & Montanari, A. (2024). Dimension free ridge regression. The Annals of Statistics, 52(6), 2879-2912. https://doi.org/10.1214/24-aos2449
Vancouver
Cheng C, Montanari A. Dimension free ridge regression. The Annals of Statistics. 2024;52(6):2879-2912. doi:10.1214/24-aos2449
BibTeX
@article{Cheng2024,
author = {Cheng, Chen and Montanari, Andrea},
title = {Dimension free ridge regression},
journal = {The Annals of Statistics},
year = {2024},
volume = {52},
number = {6},
pages = {2879--2912},
doi = {10.1214/24-aos2449}
}
EUVIMED