UAdam: Unified Adam-Type Algorithmic Framework for Nonconvex Optimization
Neural Computation · 19.08.2024 · 36 (9) · 1912-1938
DOI: 10.1162/neco_a_01692 ↗Kurz gefasst
Fachartikel in der Zeitschrift Neural Computation, online erschienen am 19.08.2024. Verfasst von Yiming Jiang und 3 weiteren Autor:innen. Beteiligte Einrichtung: Key Laboratory for Applied Statistics of MOE, School of Mathematics and Statistics, Northeast Normal University, Changchun 130024, China jiangym048@n…. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 44 Literaturangaben.
Abstract (Originalsprache)
Adam-type algorithms have become a preferred choice for optimization in the deep learning setting; however, despite their success, their convergence is still not well understood. To this end, we introduce a unified framework for Adam-type algorithms, termed UAdam. It is equipped with a general form of the second-order moment, which makes it possible to include Adam and its existing and future variants as special cases, such as NAdam, AMSGrad, AdaBound, AdaFom, and Adan. The approach is supported by a rigorous convergence analysis of UAdam in the general nonconvex stochastic setting, showing that UAdam converges to the neighborhood of stationary points with a rate of O(1/T). Furthermore, the size of the neighborhood decreases as the parameter β1 increases. Importantly, our analysis only requires the first-order momentum factor to be close enough to 1, without any restrictions on the second-order momentum factor. Theoretical results also reveal the convergence conditions of vanilla Adam, together with the selection of appropriate hyperparameters. This provides a theoretical guarantee for the analysis, applications, and further developments of the whole general class of Adam-type algorithms. Finally, several numerical experiments are provided to support our theoretical findings.
Autor:innen und Institutionen
- Key Laboratory for Applied Statistics of MOE, School of Mathematics and Statistics, Northeast Normal University, Changchun 130024, China jiangym048@nenu.edu.cn.
- Department of Mathematics, Changchun Normal University, Changchun 130032, China liujinlan@ccsfu.edu.cn.
- Key Laboratory for Applied Statistics of MOE, School of Mathematics and Statistics, Northeast Normal University, Changchun 130024, China xudp100@nenu.edu.cn.
- Department of Electrical and Electronic Engineering, Imperial College London, SW7 2AZ London, U.K. d.mandic@imperial.ac.uk.
Zitieren
APA
Jiang, Y., Liu, J., Xu, D., & Mandic, D. P. (2024). UAdam: Unified Adam-Type Algorithmic Framework for Nonconvex Optimization. Neural Computation, 36(9), 1912-1938. https://doi.org/10.1162/neco_a_01692
Vancouver
Jiang Y, Liu J, Xu D, Mandic DP. UAdam: Unified Adam-Type Algorithmic Framework for Nonconvex Optimization. Neural Computation. 2024;36(9):1912-1938. doi:10.1162/neco_a_01692
BibTeX
@article{Jiang2024,
author = {Jiang, Yiming and Liu, Jinlan and Xu, Dongpo and Mandic, Danilo P},
title = {UAdam: Unified Adam-Type Algorithmic Framework for Nonconvex Optimization},
journal = {Neural Computation},
year = {2024},
volume = {36},
number = {9},
pages = {1912--1938},
doi = {10.1162/neco_a_01692}
}
EUVIMED