Nonparametric Bellman Mappings for Value Iteration in Distributed Reinforcement Learning
IEEE Transactions on Signal Processing · 2026 · 74 · 1444-1460
DOI: 10.1109/tsp.2026.3671726 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Signal Processing, 2026. Verfasst von Yuki Akiyama und 2 weiteren Autor:innen. Die Arbeit nennt 48 Literaturangaben.
Abstract (Originalsprache)
This paper introduces novel Bellman mappings (B-Maps) for value iteration (VI) in distributed reinforcement learning (DRL), where agents are deployed over an undirected, connected graph/network with arbitrary topology—but without a centralized node, that is, a node capable of aggregating all data and performing computations. Each agent constructs a nonparametric B-Map from its private data, operating on Q-functions represented in a reproducing kernel Hilbert space, with flexibility in choosing the basis for their representation. Agents exchange their Q-function estimates only with direct neighbors, and unlike existing DRL approaches that restrict communication to Q-functions, the proposed framework also enables the transmission of basis information in the form of covariance matrices, thereby conveying additional structural details. Linear convergence rates are established for both Q-function and covariance-matrix estimates toward their consensus values, regardless of the network topology, with optimal learning rates determined by the ratio of the smallest positive eigenvalue (the graph’s Fiedler value) to the largest eigenvalue of the graph Laplacian matrix. A detailed performance analysis further shows that the proposed DRL framework effectively approximates the performance of a centralized node, had such a node existed. Numerical tests on distributed extensions of three benchmark control problems confirm the effectiveness of the proposed nonparametric B-Maps relative to state-of-the-art methods. Notably, the tests reveal a counter-intuitive outcome: although the framework involves richer information exchange—specifically, through the transmission of covariance matrices as information on the basis of a subspace—it achieves the desired performance at a lower cumulative communication cost than several state-of-the-art DRL schemes, underscoring the critical role of sharing basis information in accelerating the learning process.
Zitieren
APA
Akiyama, Y., Vu, M., & Slavakis, K. (2026). Nonparametric Bellman Mappings for Value Iteration in Distributed Reinforcement Learning. IEEE Transactions on Signal Processing, 74, 1444-1460. https://doi.org/10.1109/tsp.2026.3671726
Vancouver
Akiyama Y, Vu M, Slavakis K. Nonparametric Bellman Mappings for Value Iteration in Distributed Reinforcement Learning. IEEE Transactions on Signal Processing. 2026;74:1444-1460. doi:10.1109/tsp.2026.3671726
BibTeX
@article{Akiyama2026,
author = {Akiyama, Yuki and Vu, Minh and Slavakis, Konstantinos},
title = {Nonparametric Bellman Mappings for Value Iteration in Distributed Reinforcement Learning},
journal = {IEEE Transactions on Signal Processing},
year = {2026},
volume = {74},
pages = {1444--1460},
doi = {10.1109/tsp.2026.3671726}
}
EUVIMED