EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Mitigating the Vanishing Gradient Problem Using a Pseudo-Normalizing Method

Yun Bu iD, Wenbo Jiang iD, Gang Lu, Qiang Zhang

Entropy · 31.12.2025 · 28 (1) · 57

DOI: 10.3390/e28010057 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Entropy, online erschienen am 31.12.2025. Verfasst von Yun Bu und 3 weiteren Autor:innen. Beteiligte Einrichtung: School of Electrical Engineering and Electronic Information, Xihua University, Chengdu 610039, China. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 30 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

When training a neural network, the choice of activation function can greatly impact its performance. A function with a larger derivative may cause the coefficients of the latter layers to deviate further from the calculated direction, making deep learning more difficult to train. However, an activation function with a derivative amplitude of less than one can result in the problem of a vanishing gradient. To overcome this drawback, we propose the application of pseudo-normalization to enlarge some gradients by dividing them by the root mean square. This amplification is performed every few layers to ensure that the amplitudes are larger than one, thus avoiding the condition of vanishing gradient and preventing gradient explosion. We successfully applied this approach to several deep learning networks with hyperbolic tangent activation for image classifications. To gain a deeper understanding of the algorithm, we employed interpretability techniques to examine the network’s prediction outcomes. We discovered that, in contrast to popular networks that learn picture characteristics, the networks primarily employ the contour information of images for categorization. This suggests that our technique can be utilized in addition to other widely used algorithms.

Schlagwörter

Stichwörter der Autor:innen

backpropagation algorithmconvolutional neural networksdeep learningpseudo-normalizingvanishing gradient

Autor:innen und Institutionen

  • School of Electrical Engineering and Electronic Information, Xihua University, Chengdu 610039, China.
  • School of Sciences, Southwest Petroleum University, Chengdu 610500, China.

ORCID-Profile: Yun Bu, Wenbo Jiang

Zitieren

APA

Bu, Y., Jiang, W., Lu, G., & Zhang, Q. (2025). Mitigating the Vanishing Gradient Problem Using a Pseudo-Normalizing Method. Entropy, 28(1), 57. https://doi.org/10.3390/e28010057

Vancouver

Bu Y, Jiang W, Lu G, Zhang Q. Mitigating the Vanishing Gradient Problem Using a Pseudo-Normalizing Method. Entropy. 2025;28(1):57. doi:10.3390/e28010057

BibTeX
@article{Bu2025,
  author = {Bu, Yun and Jiang, Wenbo and Lu, Gang and Zhang, Qiang},
  title = {Mitigating the Vanishing Gradient Problem Using a Pseudo-Normalizing Method},
  journal = {Entropy},
  year = {2025},
  volume = {28},
  number = {1},
  pages = {57},
  doi = {10.3390/e28010057}
}
Kostenlos starten →