Elucidating the Theoretical Underpinnings of Surrogate Gradient Learning in Spiking Neural Networks
Neural Computation · 17.04.2025 · 37 (5) · 886-925
DOI: 10.1162/neco_a_01752 ↗Kurz gefasst
Fachartikel in der Zeitschrift Neural Computation, online erschienen am 17.04.2025. Verfasst von Julia Gygax und 1 weiteren Autor:innen. Beteiligte Einrichtung: Friedrich Miescher Institute for Biomedical Research and Faculty of Science, University of Basel, Basel 4056, Switzerland julia.gygax@fmi.ch. Hauptthemen laut MEDLINE: Neural Networks, Computer, Action Potentials, Neurons, Models, Neurological. Die Arbeit wurde bisher 17-mal zitiert (Crossref) und nennt 60 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Swiss National Science Foundation.
Abstract (Originalsprache)
Training spiking neural networks to approximate universal functions is essential for studying information processing in the brain and for neuromorphic computing. Yet the binary nature of spikes poses a challenge for direct gradient-based training. Surrogate gradients have been empirically successful in circumventing this problem, but their theoretical foundation remains elusive. Here, we investigate the relation of surrogate gradients to two theoretically well-founded approaches. On the one hand, we consider smoothed probabilistic models, which, due to the lack of support for automatic differentiation, are impractical for training multilayer spiking neural networks but provide derivatives equivalent to surrogate gradients for single neurons. On the other hand, we investigate stochastic automatic differentiation, which is compatible with discrete randomness but has not yet been used to train spiking neural networks. We find that the latter gives surrogate gradients a theoretical basis in stochastic spiking neural networks, where the surrogate derivative matches the derivative of the neuronal escape noise function. This finding supports the effectiveness of surrogate gradients in practice and suggests their suitability for stochastic spiking neural networks. However, surrogate gradients are generally not gradients of a surrogate loss despite their relation to stochastic automatic differentiation. Nevertheless, we empirically confirm the effectiveness of surrogate gradients in stochastic multilayer spiking neural networks and discuss their relation to deterministic networks as a special case. Our work gives theoretical support to surrogate gradients and the choice of a suitable surrogate derivative in stochastic spiking neural networks.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Friedrich Miescher Institute for Biomedical Research and Faculty of Science, University of Basel, Basel 4056, Switzerland julia.gygax@fmi.ch.
- Friedrich Miescher Institute for Biomedical Research and Faculty of Science, University of Basel, Basel 4056, Switzerland friedemann.zenke@fmi.ch.
Förderung
- Swiss National Science Foundation
Zitieren
APA
Gygax, J., & Zenke, F. (2025). Elucidating the Theoretical Underpinnings of Surrogate Gradient Learning in Spiking Neural Networks. Neural Computation, 37(5), 886-925. https://doi.org/10.1162/neco_a_01752
Vancouver
Gygax J, Zenke F. Elucidating the Theoretical Underpinnings of Surrogate Gradient Learning in Spiking Neural Networks. Neural Computation. 2025;37(5):886-925. doi:10.1162/neco_a_01752
BibTeX
@article{Gygax2025,
author = {Gygax, Julia and Zenke, Friedemann},
title = {Elucidating the Theoretical Underpinnings of Surrogate Gradient Learning in Spiking Neural Networks},
journal = {Neural Computation},
year = {2025},
volume = {37},
number = {5},
pages = {886--925},
doi = {10.1162/neco_a_01752}
}
EUVIMED