Efficient Deep Reinforcement Learning with Predictive Processing Proximal Policy Optimization
Neurons, Behavior, Data analysis, and Theory · 2024
DOI: 10.51628/001c.123366 ↗Kurz gefasst
Fachartikel in der Zeitschrift Neurons, Behavior, Data analysis, and Theory, 2024. Verfasst von Burcu Küçükoğlu und 5 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Advances in reinforcement learning (RL) often rely on massive compute resources and remain notoriously sample inefficient. In contrast, the human brain is able to efficiently learn effective control strategies using limited resources. This raises the question whether insights from neuroscience can be used to improve current RL methods. Predictive processing is a popular theoretical framework which maintains that the human brain is actively seeking to minimize surprise. We show that recurrent neural networks which predict their own sensory states can be leveraged to minimise surprise, yielding substantial gains in cumulative reward. Specifically, we present the Predictive Processing Proximal Policy Optimization (P4O) agent; an actor-critic reinforcement learning agent that applies predictive processing to a recurrent variant of the PPO algorithm by integrating a world model in its hidden state. Even without hyperparameter tuning, P4O significantly outperforms a baseline recurrent variant of the PPO algorithm on multiple Atari games using a single GPU. It also outperforms other state-of-the-art agents given the same wall-clock time and exceeds human gamer performance on multiple games including Seaquest, which is a particularly challenging environment in the Atari domain. Altogether, our work underscores how insights from the field of neuroscience may support the development of more capable and efficient artificial agents.
Zitieren
APA
Küçükoğlu, B., Borkent, W., Rueckauer, B., Ahmad, N., Güçlü, U., & Gerven, M. V. (2024). Efficient Deep Reinforcement Learning with Predictive Processing Proximal Policy Optimization. Neurons, Behavior, Data analysis, and Theory. https://doi.org/10.51628/001c.123366
Vancouver
Küçükoğlu B, Borkent W, Rueckauer B, Ahmad N, Güçlü U, Gerven MV. Efficient Deep Reinforcement Learning with Predictive Processing Proximal Policy Optimization. Neurons, Behavior, Data analysis, and Theory. 2024. doi:10.51628/001c.123366
BibTeX
@article{Kkolu2024,
author = {Küçükoğlu, Burcu and Borkent, Walraaf and Rueckauer, Bodo and Ahmad, Nasir and Güçlü, Umut and Gerven, Marcel van},
title = {Efficient Deep Reinforcement Learning with Predictive Processing Proximal Policy Optimization},
journal = {Neurons, Behavior, Data analysis, and Theory},
year = {2024},
doi = {10.51628/001c.123366}
}
EUVIMED