Toward Enhancing RMSProp With Forward-Looking Gradient Updates for Complex Loss Landscapes
Neural Computation · 20.05.2026 · 38 (6) · 1062-1089
DOI: 10.1162/neco.a.1514 ↗Kurz gefasst
Fachartikel in der Zeitschrift Neural Computation, online erschienen am 20.05.2026. Verfasst von Rafał Wolniak und 1 weiteren Autor:innen. Beteiligte Einrichtung: Gdańsk University of Technology, Faculty of Electronics, Telecommunications and Informatics, Multimedia Systems Department and Audio Acoustics Labora…. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 52 Literaturangaben.
Abstract (Originalsprache)
This letter introduces a novel algorithm for training deep neural networks with many nonlinear layers. Our method uses an approximated integrated gradient that is averaged over the range of the weight update to more accurately capture the loss change resulting from parameter updates. Unlike standard gradients, this average gradient improves learning efficiency in certain scenarios. We incorporate the approximated average gradient into RMSProp and compare the resulting algorithm to conventional RMSProp and Adam. We evaluate the approach on deep models lacking skip connections, such as those with many nonlinear activations and no residual structure, where traditional methods typically encounter difficulties. These models that focus on extracting high-order features create a loss landscape more akin to that of a biological brain. Our method requires significantly fewer iterations to reach a target training loss on MNIST, Fashion MNIST, and IMDb benchmarks for both convolutional and fully connected architectures across different initialization schemes. While our approach incurs moderately higher computational and memory costs compared to standard RMSProp, its performance on shallow models remains comparable. Nevertheless, our main contributions are (1) introducing the average gradient concept as an efficient alternative to computing high–order derivatives, (2) offering a novel factorization formula for approximating the average gradient, accompanied by a formal derivation, and (3) showing an example algorithm that leverages this formula to enhance the efficiency of RMSProp for some models, as validated by our evaluation.
Autor:innen und Institutionen
- Gdańsk University of Technology, Faculty of Electronics, Telecommunications and Informatics, Multimedia Systems Department and Audio Acoustics Laboratory, Gabriela Narutowicza 11/12, Gdańsk 80-233, Poland rafal.wolniak@…
- Gdańsk University of Technology, Faculty of Electronics, Telecommunications and Informatics, Multimedia Systems Department and Audio Acoustics Laboratory, Gabriela Narutowicza 11/12, Gdańsk 80-233, Poland bozena.kostek@…
Zitieren
APA
Wolniak, R., & Kostek, B. (2026). Toward Enhancing RMSProp With Forward-Looking Gradient Updates for Complex Loss Landscapes. Neural Computation, 38(6), 1062-1089. https://doi.org/10.1162/neco.a.1514
Vancouver
Wolniak R, Kostek B. Toward Enhancing RMSProp With Forward-Looking Gradient Updates for Complex Loss Landscapes. Neural Computation. 2026;38(6):1062-1089. doi:10.1162/neco.a.1514
BibTeX
@article{Wolniak2026,
author = {Wolniak, Rafał and Kostek, Bożena},
title = {Toward Enhancing RMSProp With Forward-Looking Gradient Updates for Complex Loss Landscapes},
journal = {Neural Computation},
year = {2026},
volume = {38},
number = {6},
pages = {1062--1089},
doi = {10.1162/neco.a.1514}
}
EUVIMED