A Local Convergence Theory for the Stochastic Gradient Descent Method in Non-Convex Optimization with Non-Isolated Local Minima
Journal of Machine Learning · 06.06.2023 · 2 (2) · 138-160
DOI: 10.4208/jml.v2.n2.p138 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Machine Learning, online erschienen am 06.06.2023. Verfasst von Taehee Ko und 1 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 41 Literaturangaben.
Abstract (Originalsprache)
Loss functions with non-isolated minima have emerged in several machine-learning problems, creating a gap between theoretical predictions and observations in practice. In this paper, we formulate a new type of local convexity condition that is suitable to describe the behavior of loss functions near non-isolated minima. We show that such a condition is general enough to encompass many existing conditions. In addition, we study the local convergence of the stochastic gradient descent (SGD) method under this mild condition by adopting the notion of stochastic stability. In the convergence analysis, we establish concentration inequalities for the iterates in SGD, which can be used to interpret the empirical observation from some practical training results.
Zitieren
APA
Ko, T., & Li, X. (2023). A Local Convergence Theory for the Stochastic Gradient Descent Method in Non-Convex Optimization with Non-Isolated Local Minima. Journal of Machine Learning, 2(2), 138-160. https://doi.org/10.4208/jml.v2.n2.p138
Vancouver
Ko T, Li X. A Local Convergence Theory for the Stochastic Gradient Descent Method in Non-Convex Optimization with Non-Isolated Local Minima. Journal of Machine Learning. 2023;2(2):138-160. doi:10.4208/jml.v2.n2.p138
BibTeX
@article{Ko2023,
author = {Ko, Taehee and Li, Xiantao},
title = {A Local Convergence Theory for the Stochastic Gradient Descent Method in Non-Convex Optimization with Non-Isolated Local Minima},
journal = {Journal of Machine Learning},
year = {2023},
volume = {2},
number = {2},
pages = {138--160},
doi = {10.4208/jml.v2.n2.p138}
}
EUVIMED