Emergence of belief-like representations through reinforcement learning
PLOS Computational Biology · 2023
DOI: 10.1371/journal.pcbi.1011067 ↗Kurz gefasst
Fachartikel in der Zeitschrift PLOS Computational Biology, 2023. Verfasst von Jay A Hennig und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Psychology, Harvard University, Cambridge, Massachusetts, United States of America. Hauptthemen laut MEDLINE: Learning, Reinforcement, Psychology. Die Arbeit wurde bisher 44-mal zitiert (Crossref) und nennt 49 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NINDS NIH HHS, National Institutes of Health, Air Force Research Laboratory.
Abstract (Originalsprache)
To behave adaptively, animals must learn to predict future reward, or value. To do this, animals are thought to learn reward predictions using reinforcement learning. However, in contrast to classical models, animals must learn to estimate value using only incomplete state information. Previous work suggests that animals estimate value in partially observable tasks by first forming “beliefs”—optimal Bayesian estimates of the hidden states in the task. Although this is one way to solve the problem of partial observability, it is not the only way, nor is it the most computationally scalable solution in complex, real-world environments. Here we show that a recurrent neural network (RNN) can learn to estimate value directly from observations, generating reward prediction errors that resemble those observed experimentally, without any explicit objective of estimating beliefs. We integrate statistical, functional, and dynamical systems perspectives on beliefs to show that the RNN’s learned representation encodes belief information, but only when the RNN’s capacity is sufficiently large. These results illustrate how animals can estimate value in tasks without explicitly estimating beliefs, yielding a representation useful for systems with limited capacity.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Department of Psychology, Harvard University, Cambridge, Massachusetts, United States of America.
- Center for Brain Science, Harvard University, Cambridge, Massachusetts, United States of America.
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, Massachusetts, United States of America.
- Program in Speech and Hearing Bioscience and Technology, Harvard Medical School, Boston, Massachusetts, USA.
- Future Research Department, Toyota Research Institute of North America, Toyota Motor North America, Ann Arbor, Michigan, United States of America.
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, California, United States of America.
ORCID-Profile: Jay A Hennig
Förderung
- NINDS NIH HHS
- National Institutes of Health
- Air Force Research Laboratory
Zitieren
APA
Hennig, J. A., Romero Pinto, S. A., Yamaguchi, T., Linderman, S. W., Uchida, N., & Gershman, S. J. (2023). Emergence of belief-like representations through reinforcement learning. PLOS Computational Biology. https://doi.org/10.1371/journal.pcbi.1011067
Vancouver
Hennig JA, Romero Pinto SA, Yamaguchi T, Linderman SW, Uchida N, Gershman SJ. Emergence of belief-like representations through reinforcement learning. PLOS Computational Biology. 2023. doi:10.1371/journal.pcbi.1011067
BibTeX
@article{Hennig2023,
author = {Hennig, Jay A and Romero Pinto, Sandra A and Yamaguchi, Takahiro and Linderman, Scott W and Uchida, Naoshige and Gershman, Samuel J},
title = {Emergence of belief-like representations through reinforcement learning},
journal = {PLOS Computational Biology},
year = {2023},
doi = {10.1371/journal.pcbi.1011067}
}
EUVIMED