Adaptive Biased Stochastic Optimization
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-04 · 47 (4) · 3067-3078
DOI: 10.1109/tpami.2025.3528193 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-04. Verfasst von Zhuang Yang. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 53 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Natural Science Foundation of Jiangsu Province in China, Project Funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions.
Abstract (Originalsprache)
This work develops and analyzes a class of adaptive biased stochastic optimization (ABSO) algorithms from the perspective of the GEneralized Adaptive gRadient (GEAR) method that contains Adam, AdaGrad, RMSProp, etc. Particularly, two preferred biased stochastic optimization (BSO) algorithms, the biased stochastic variance reduction gradient (BSVRG) algorithm and the stochastic recursive gradient algorithm (SARAH), equipped with GEAR, are first considered in this work, leading to two ABSO algorithms: BSVRG-GEAR and SARAH-GEAR. We present a uniform analysis of ABSO algorithms for minimizing strongly convex (SC) and Polyak-Łojasiewicz (PŁ) composite objective functions. Second, we also use our framework to develop another novel BSO algorithm, adaptive biased stochastic conjugate gradient (coined BSCG-GEAR), which achieves the well-known oracle complexity. Specifically, under mild conditions, we prove that the resulting ABSO algorithms attain a linear convergence rate on both PŁ and SC cases. Moreover, we show that the complexity of the resulting ABSO algorithms is comparable to that of advanced stochastic gradient-based algorithms. Finally, we demonstrate the empirical superiority and the numerical stability of the resulting ABSO algorithms by conducting numerical experiments on different applications of machine learning.
Förderung
- National Natural Science Foundation of China
- Natural Science Foundation of Jiangsu Province in China
- Project Funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions
Zitieren
APA
Yang, Z. (2025). Adaptive Biased Stochastic Optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(4), 3067-3078. https://doi.org/10.1109/tpami.2025.3528193
Vancouver
Yang Z. Adaptive Biased Stochastic Optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(4):3067-3078. doi:10.1109/tpami.2025.3528193
BibTeX
@article{Yang2025,
author = {Yang, Zhuang},
title = {Adaptive Biased Stochastic Optimization},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2025},
volume = {47},
number = {4},
pages = {3067--3078},
doi = {10.1109/tpami.2025.3528193}
}
EUVIMED