EUVIMEDEUVIMED entdecken
Crossref

Towards Understanding Convergence and Generalization of AdamW

Pan Zhou, Xingyu Xie, Zhouchen Lin, Shuicheng Yan

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-09 · 46 (9) · 6486-6493

DOI: 10.1109/tpami.2024.3382294 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-09. Verfasst von Pan Zhou und 3 weiteren Autor:innen. Die Arbeit wurde bisher 369-mal zitiert (Crossref) und nennt 64 Literaturangaben. Gefördert von: Ministry of Education - Singapore, NSF China, PCL, China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

AdamW modifies Adam by adding a decoupled weight decay to decay network weights per training iteration. For adaptive algorithms, this decoupled weight decay does not affect specific optimization steps, and differs from the widely used l2-regularizer which changes optimization steps via changing the first- and second-order gradient moments. Despite its great practical success, for AdamW, its convergence behavior and generalization improvement over Adam and l2-regularized Adam ( l2-Adam) remain absent yet. To solve this issue, we prove the convergence of AdamW and justify its generalization advantages over Adam and l2-Adam. Specifically, AdamW provably converges but minimizes a dynamically regularized loss that combines vanilla loss and a dynamical regularization induced by decoupled weight decay, thus yielding different behaviors with Adam and l2-Adam. Moreover, on both general nonconvex problems and PŁ-conditioned problems, we establish stochastic gradient complexity of AdamW to find a stationary point. Such complexity is also applicable to Adam and l2-Adam, and improves their previously known complexity, especially for over-parametrized networks. Besides, we prove that AdamW enjoys smaller generalization errors than Adam and l2-Adam from the Bayesian posterior aspect. This result, for the first time, explicitly reveals the benefits of decoupled weight decay in AdamW. Experimental results validate our theory.

Förderung

  • Ministry of Education - Singapore
  • NSF China
  • PCL, China

Zitieren

APA

Zhou, P., Xie, X., Lin, Z., & Yan, S. (2024). Towards Understanding Convergence and Generalization of AdamW. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(9), 6486-6493. https://doi.org/10.1109/tpami.2024.3382294

Vancouver

Zhou P, Xie X, Lin Z, Yan S. Towards Understanding Convergence and Generalization of AdamW. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(9):6486-6493. doi:10.1109/tpami.2024.3382294

BibTeX
@article{Zhou2024,
  author = {Zhou, Pan and Xie, Xingyu and Lin, Zhouchen and Yan, Shuicheng},
  title = {Towards Understanding Convergence and Generalization of AdamW},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2024},
  volume = {46},
  number = {9},
  pages = {6486--6493},
  doi = {10.1109/tpami.2024.3382294}
}
Kostenlos starten →