PID controller‐based adaptive gradient optimizer for deep neural networks
IET Control Theory & Applications · 2023-10 · 17 (15) · 2032-2037
DOI: 10.1049/cth2.12404 ↗Kurz gefasst
Fachartikel in der Zeitschrift IET Control Theory & Applications, online erschienen am 26.11.2022. Verfasst von Mingjun Dai und 4 weiteren Autor:innen. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: National Natural Science Foundation of China.
Abstract (Originalsprache)
Due to improper selection of gradient update direction or learning rate, SGD optimization algorithms for deep learning suffer from oscillation and slow convergence. Although Adam algorithm can adaptively adjust the update direction and learning rate at the same time, it still has the overshoot phenomenon, and hence suffers from wasting computing resources and slow convergence. In this work, the PID controller from the feedback control area is borrowed to re‐express the adaptive optimization algorithm (the Adam optimization algorithm is derived into the integral I component form) of deep learning. In order to alleviate the overshoot phenomenon and hence speed up the convergence of Adam, a complete adaptive PID optimizer (adaptive‐PID) is proposed by incorporating the proportional P and derivative D component. Extensive experiments on standard data sets verify that the proposed adaptive‐PID algorithm significantly outperforms Adam algorithm in terms of convergence rate and accuracy.
Förderung
- National Natural Science Foundation of China
Zitieren
APA
Dai, M., Zhang, Z., Lai, X., Lin, X., & Wang, H. (2023). PID controller‐based adaptive gradient optimizer for deep neural networks. IET Control Theory & Applications, 17(15), 2032-2037. https://doi.org/10.1049/cth2.12404
Vancouver
Dai M, Zhang Z, Lai X, Lin X, Wang H. PID controller‐based adaptive gradient optimizer for deep neural networks. IET Control Theory & Applications. 2023;17(15):2032-2037. doi:10.1049/cth2.12404
BibTeX
@article{Dai2023,
author = {Dai, Mingjun and Zhang, Zelong and Lai, Xiong and Lin, Xiaohui and Wang, Hui},
title = {PID controller‐based adaptive gradient optimizer for deep neural networks},
journal = {IET Control Theory & Applications},
year = {2023},
volume = {17},
number = {15},
pages = {2032--2037},
doi = {10.1049/cth2.12404}
}
EUVIMED