EUVIMEDOpdag EUVIMED
Open access · CC BY 4.0

New efficient accelerated and stochastic gradient descent algorithms based on locally Lipschitz gradient constants

Nguyen Phung Hai Chung, Pham Thi Hoai, Hoang Van Chung

Journal of Convex Analysis · 2027 · 34 (1) · 271-293

DOI: 10.68381/jca34020 ↗

At a glance

Journal article in Journal of Convex Analysis, 2027. Written by Nguyen Phung Hai Chung and 2 co-authors. 20 references. The full text is freely available (open access, CC BY 4.0).

AbstractBibliografiske oplysningerCiteRelated studies

Abstract (original language)

This paper revisits the recently introduced stepsize in the gradient descent scheme called NGD, proposed by Hoai et al. We first investigate the NGD stepsize in conjunction with two well-known acceleration techniques: the Heavy Ball method and Nesterov's method. In the convex setting of unconstrained nonlinear optimization problems, we establish the ergodic convergence of the iterates generated by the accelerated versions of NGD at a sublinear rate. We introduce stochastic versions of the proposed accelerated algorithms and analyze their convergence in nonconvex settings. Although our proposed algorithms require global Lipschitz continuity of the gradient, we do not utilize the global Lipschitz constant during computations. Instead, we leverage information about local Lipschitz constants derived from previous iterations. Numerical experiments on various practical machine learning and deep learning problems demonstrate the efficiency of our proposed methods compared to the recent ones.

Cite

APA

Chung, N. P. H., Hoai, P. T., & Chung, H. V. (2027). New efficient accelerated and stochastic gradient descent algorithms based on locally Lipschitz gradient constants. Journal of Convex Analysis, 34(1), 271-293. https://doi.org/10.68381/jca34020

Vancouver

Chung NPH, Hoai PT, Chung HV. New efficient accelerated and stochastic gradient descent algorithms based on locally Lipschitz gradient constants. Journal of Convex Analysis. 2027;34(1):271-293. doi:10.68381/jca34020

BibTeX
@article{Chung2027,
  author = {Chung, Nguyen Phung Hai and Hoai, Pham Thi and Chung, Hoang Van},
  title = {New efficient accelerated and stochastic gradient descent algorithms based on locally Lipschitz gradient constants},
  journal = {Journal of Convex Analysis},
  year = {2027},
  volume = {34},
  number = {1},
  pages = {271--293},
  doi = {10.68381/jca34020}
}
Start for free →