EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Toward robust and scalable deep spiking reinforcement learning

Mahmoud Akl, Deniz Ergene, Florian Walter, Alois Knoll

Frontiers in Neurorobotics · 20.01.2023 · 16

DOI: 10.3389/fnbot.2022.1075647 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Frontiers in Neurorobotics, online erschienen am 20.01.2023. Verfasst von Mahmoud Akl und 3 weiteren Autor:innen. Beteiligte Einrichtung: Chair of Robotics, Artificial Intelligence and Embedded Systems, TUM School of Computation, Information and Technology, Technische Universität Münche…. Die Arbeit wurde bisher 29-mal zitiert (Crossref) und nennt 54 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Deep reinforcement learning (DRL) combines reinforcement learning algorithms with deep neural networks (DNNs). Spiking neural networks (SNNs) have been shown to be a biologically plausible and energy efficient alternative to DNNs. Since the introduction of surrogate gradient approaches that allowed to overcome the discontinuity in the spike function, SNNs can now be trained with the backpropagation through time (BPTT) algorithm. While largely explored on supervised learning problems, little work has been done on investigating the use of SNNs as function approximators in DRL. Here we show how SNNs can be applied to different DRL algorithms like Deep Q-Network (DQN) and Twin-Delayed Deep Deteministic Policy Gradient (TD3) for discrete and continuous action space environments, respectively. We found that SNNs are sensitive to the additional hyperparameters introduced by spiking neuron models like current and voltage decay factors, firing thresholds, and that extensive hyperparameter tuning is inevitable. However, we show that increasing the simulation time of SNNs, as well as applying a two-neuron encoding to the input observations helps reduce the sensitivity to the membrane parameters. Furthermore, we show that randomizing the membrane parameters, instead of selecting uniform values for all neurons, has stabilizing effects on the training. We conclude that SNNs can be utilized for learning complex continuous control problems with state-of-the-art DRL algorithms. While the training complexity increases, the resulting SNNs can be directly executed on neuromorphic processors and potentially benefit from their high energy efficiency.

Schlagwörter

Stichwörter der Autor:innen

continuous controldeep reinforcement learning (Deep RL)hyperparameter tuningreinforcement learningspiking neural network (SNN)

Autor:innen und Institutionen

  • Chair of Robotics, Artificial Intelligence and Embedded Systems, TUM School of Computation, Information and Technology, Technische Universität München, Munich, Germany.

Zitieren

APA

Akl, M., Ergene, D., Walter, F., & Knoll, A. (2023). Toward robust and scalable deep spiking reinforcement learning. Frontiers in Neurorobotics, 16. https://doi.org/10.3389/fnbot.2022.1075647

Vancouver

Akl M, Ergene D, Walter F, Knoll A. Toward robust and scalable deep spiking reinforcement learning. Frontiers in Neurorobotics. 2023;16. doi:10.3389/fnbot.2022.1075647

BibTeX
@article{Akl2023,
  author = {Akl, Mahmoud and Ergene, Deniz and Walter, Florian and Knoll, Alois},
  title = {Toward robust and scalable deep spiking reinforcement learning},
  journal = {Frontiers in Neurorobotics},
  year = {2023},
  volume = {16},
  doi = {10.3389/fnbot.2022.1075647}
}
Kostenlos starten →