EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Online reinforcement learning of state representation in recurrent network supported by the power of random feedback and biological constraints

Takayuki Tsurumi, Ayaka Kato iD, Arvind Kumar iD, Kenji Morita iD

eLife · 2025

DOI: 10.7554/elife.104101 ↗

Kurz gefasst

Fachartikel in der Zeitschrift eLife, 2025. Verfasst von Takayuki Tsurumi und 3 weiteren Autor:innen. Beteiligte Einrichtung: Physical and Health Education, Graduate School of Education, The University of Tokyo, Tokyo, Japan. Hauptthemen laut MEDLINE: Neural Networks, Computer, Reinforcement, Psychology, Learning. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 143 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Japan Society for the Promotion of Science, Naito Foundation, Digital Futures.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Representation of external and internal states in the brain plays a critical role in enabling suitable behavior. Recent studies suggest that state representation and state value can be simultaneously learned through Temporal-Difference-Reinforcement-Learning (TDRL) and Backpropagation-Through-Time (BPTT) in recurrent neural networks (RNNs) and their readout. However, neural implementation of such learning remains unclear as BPTT requires offline update using transported downstream weights, which is suggested to be biologically implausible. We demonstrate that simple online training of RNNs using TD reward prediction error and random feedback, without additional memory or eligibility trace, can still learn the structure of tasks with cue–reward delay and timing variability. This is because TD learning itself is a solution for temporal credit assignment, and feedback alignment, a mechanism originally proposed for supervised learning, enables gradient approximation without weight transport. Furthermore, we show that biologically constraining downstream weights and random feedback to be non-negative not only preserves learning but may even enhance it because the non-negative constraint ensures loose alignment—allowing the downstream and feedback weights to roughly align from the beginning. These results provide insights into the neural mechanisms underlying the learning of state representation and value, highlighting the potential of random feedback and biological constraints.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Neural Networks, ComputerReinforcement, PsychologyLearningHumansBrainRewardFeedback

Stichwörter der Autor:innen

biological constraintscorticostriataldopaminefeedback alignmentneurosciencenonereinforcement learningstate representation

Autor:innen und Institutionen

  • Physical and Health Education, Graduate School of Education, The University of Tokyo, Tokyo, Japan.
  • Theoretical Sciences Visiting Program, Okinawa Institute of Science and Technology, Okinawa, Japan.
  • Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, United States.
  • Division of Computational Science and Technology, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, Stockholm, Sweden.
  • Science for Life Laboratory, Solna, Stockholm, Sweden.
  • International Research Center for Neurointelligence (WPI-IRCN), The University of Tokyo, Tokyo, Japan.

ORCID-Profile: Ayaka Kato, Arvind Kumar, Kenji Morita

Förderung

  • Japan Society for the Promotion of Science
  • Naito Foundation
  • Digital Futures
  • StratNeuro SRA

Zitieren

APA

Tsurumi, T., Kato, A., Kumar, A., & Morita, K. (2025). Online reinforcement learning of state representation in recurrent network supported by the power of random feedback and biological constraints. eLife. https://doi.org/10.7554/elife.104101

Vancouver

Tsurumi T, Kato A, Kumar A, Morita K. Online reinforcement learning of state representation in recurrent network supported by the power of random feedback and biological constraints. eLife. 2025. doi:10.7554/elife.104101

BibTeX
@article{Tsurumi2025,
  author = {Tsurumi, Takayuki and Kato, Ayaka and Kumar, Arvind and Morita, Kenji},
  title = {Online reinforcement learning of state representation in recurrent network supported by the power of random feedback and biological constraints},
  journal = {eLife},
  year = {2025},
  doi = {10.7554/elife.104101}
}
Kostenlos starten →