EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

A Comparative Analysis of Activation Functions in Neural Networks: Robustness to Input Noise

Seyed Zakaria Hashimi, Mohammad Ehsan Sorosh

Integrated Journal for Research in Arts and Humanities · 09.11.2026 · 6 (6) · 50-59

DOI: 10.55544/ijrah.6.6.6 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Integrated Journal for Research in Arts and Humanities, online erschienen am 09.11.2026. Verfasst von Seyed Zakaria Hashimi und 1 weiteren Autor:innen. Die Arbeit nennt 24 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Activation functions (AFs) are usually compared through properties such as parametric nature, monotonicity, smoothness and boundedness, whereas their behavior under perturbed inputs has received little systematic attention. This paper studies thirteen widely used activation functions from five categories (sigmoid/hyperbolic tangent, ReLU-based, exponential-unit-based, learnable/adaptive and miscellaneous) from the viewpoint of robustness to input noise. First, six analytical descriptors are derived and numerically verified for every function: the Lipschitz constant, the asymptotic slope on each side (saturation), the size of the derivative discontinuity at zero, the smoothness class, monotonicity, and the mean output for a standard normal input. This audit corrects several values that are often quoted; for example, the Lipschitz constant of SELU is 1.758, and ELU and CELU have L = 1 despite their unbounded output. Second, we test whether these descriptors predict robustness in practice. A two-hidden-layer MLP was trained with each activation (10 random seeds) on the 8×8 handwritten-digits dataset, and test accuracy was measured under additive Gaussian input noise with σ from 0 to 1. The Lipschitz constant of the activation did not predict noise robustness (Spearman ρ = 0.21, p = 0.50), and none of the other descriptors was significantly related to robustness either (|ρ| ≤ 0.44, all p ≥ 0.13). Contrary to the initial hypothesis, saturating functions (sigmoid, tanh, softsign) were not the most robust: ReLU, Leaky ReLU and GELU kept the highest accuracy under noise (mean accuracy over noisy conditions 71.2–71.5% versus 68.6–69.1%), although the saturating functions did produce the lowest sensitivity of the output logits. We conclude that a single-neuron analysis is not sufficient to rank activation functions by noise robustness, and we discuss network-level factors that are likely to matter. The results are limited to a small dataset and a shallow network.

Zitieren

APA

Hashimi, S. Z., & Sorosh, M. E. (2026). A Comparative Analysis of Activation Functions in Neural Networks: Robustness to Input Noise. Integrated Journal for Research in Arts and Humanities, 6(6), 50-59. https://doi.org/10.55544/ijrah.6.6.6

Vancouver

Hashimi SZ, Sorosh ME. A Comparative Analysis of Activation Functions in Neural Networks: Robustness to Input Noise. Integrated Journal for Research in Arts and Humanities. 2026;6(6):50-59. doi:10.55544/ijrah.6.6.6

BibTeX
@article{Hashimi2026,
  author = {Hashimi, Seyed Zakaria and Sorosh, Mohammad Ehsan},
  title = {A Comparative Analysis of Activation Functions in Neural Networks: Robustness to Input Noise},
  journal = {Integrated Journal for Research in Arts and Humanities},
  year = {2026},
  volume = {6},
  number = {6},
  pages = {50--59},
  doi = {10.55544/ijrah.6.6.6}
}
Kostenlos starten →