EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Hitting the High-dimensional notes: an ODE for SGD learning dynamics on GLMs and multi-index models

Elizabeth Collins-Woodfin, Courtney Paquette, Elliot Paquette, Inbar Seroussi

Information and Inference: A Journal of the IMA · 20.09.2024 · 13 (4)

DOI: 10.1093/imaiai/iaae028 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Information and Inference: A Journal of the IMA, online erschienen am 21.10.2024. Verfasst von Elizabeth Collins-Woodfin und 3 weiteren Autor:innen. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 55 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Canadian Institute for Advanced Research, Quebec AI Institute, Sloan Research Fellow in Computer Science.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We analyze the dynamics of streaming stochastic gradient descent (SGD) in the high-dimensional limit when applied to generalized linear models and multi-index models (e.g. logistic regression, phase retrieval) with general data-covariance. In particular, we demonstrate a deterministic equivalent of SGD in the form of a system of ordinary differential equations that describes a wide class of statistics, such as the risk and other measures of sub-optimality. This equivalence holds with overwhelming probability when the model parameter count grows proportionally to the number of data. This framework allows us to obtain learning rate thresholds for the stability of SGD as well as convergence guarantees. In addition to the deterministic equivalent, we introduce an SDE with a simplified diffusion coefficient (homogenized SGD), which allows us to analyze the dynamics of general statistics of SGD iterates. Finally, we illustrate this theory on some standard examples and show numerical simulations, which give an excellent match to the theory.

Förderung

  • Canadian Institute for Advanced Research
  • Quebec AI Institute
  • Sloan Research Fellow in Computer Science
  • Natural Science and Engineering Research Council
  • Israel Science Foundation

Zitieren

APA

Collins-Woodfin, E., Paquette, C., Paquette, E., & Seroussi, I. (2024). Hitting the High-dimensional notes: an ODE for SGD learning dynamics on GLMs and multi-index models. Information and Inference: A Journal of the IMA, 13(4). https://doi.org/10.1093/imaiai/iaae028

Vancouver

Collins-Woodfin E, Paquette C, Paquette E, Seroussi I. Hitting the High-dimensional notes: an ODE for SGD learning dynamics on GLMs and multi-index models. Information and Inference: A Journal of the IMA. 2024;13(4). doi:10.1093/imaiai/iaae028

BibTeX
@article{CollinsWoodfin2024,
  author = {Collins-Woodfin, Elizabeth and Paquette, Courtney and Paquette, Elliot and Seroussi, Inbar},
  title = {Hitting the High-dimensional notes: an ODE for SGD learning dynamics on GLMs and multi-index models},
  journal = {Information and Inference: A Journal of the IMA},
  year = {2024},
  volume = {13},
  number = {4},
  doi = {10.1093/imaiai/iaae028}
}
Kostenlos starten →