ADAPTIVE RECTIFIED LINEAR UNIT FOR DEEP NEURAL NETWORK ARCHITECTURE
Tạp chí Khoa học Lạc Hồng · 25.06.2026 · 1 (26) · 56-62
DOI: 10.61591/jslhu.26.1075 ↗Kurz gefasst
Fachartikel in der Zeitschrift Tạp chí Khoa học Lạc Hồng, online erschienen am 25.06.2026. Verfasst von Do Trung Dung.
Abstract (Originalsprache)
Activation function is a crucial factor in many deep neural network structures as it helps to boost the generalization for learning more complex mapping functions between inputs and outputs. Many activation functions can be used to build a neural network model including Rectified Linear Unit and its extended forms. By setting negative values to zeros, ReLU can introduce sparsity representation and mimic nonlinearity during training. Even though ReLU works well in creating almost Deep Neural Network structures, it still faces the problem of dying neurons especially when weights are randomly initialized. This paper introduces a novel method to activate neurons in a Deep Neural Network layer. Instead of assigning the negative values to zero, this paper automatically calculates the distribution of positive and negative values in each layer before performing the activation function. According to the distribution, the values with a higher distribution are then suppressed. Experimental results are performed with a simple Deep Neural Network structure on some public datasets such as MNIST, CIFAR10, and CIFAR100 showing that the proposed method obtains much more stable convergence results compared to the ReLU. Thus, it can be used as the default activation function when training a Deep Neural Network model.
Zitieren
APA
Dung, D. T. (2026). ADAPTIVE RECTIFIED LINEAR UNIT FOR DEEP NEURAL NETWORK ARCHITECTURE. Tạp chí Khoa học Lạc Hồng, 1(26), 56-62. https://doi.org/10.61591/jslhu.26.1075
Vancouver
Dung DT. ADAPTIVE RECTIFIED LINEAR UNIT FOR DEEP NEURAL NETWORK ARCHITECTURE. Tạp chí Khoa học Lạc Hồng. 2026;1(26):56-62. doi:10.61591/jslhu.26.1075
BibTeX
@article{Dung2026,
author = {Dung, Do Trung},
title = {ADAPTIVE RECTIFIED LINEAR UNIT FOR DEEP NEURAL NETWORK ARCHITECTURE},
journal = {Tạp chí Khoa học Lạc Hồng},
year = {2026},
volume = {1},
number = {26},
pages = {56--62},
doi = {10.61591/jslhu.26.1075}
}
EUVIMED