Multi-Timescale Ensemble $Q$-Learning for Markov Decision Process Policy Optimization
IEEE Transactions on Signal Processing · 2024 · 72 · 1427-1442
DOI: 10.1109/tsp.2024.3372699 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Signal Processing, 2024. Verfasst von Talha Bozkus und 1 weiteren Autor:innen. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 52 Literaturangaben. Gefördert von: ARO, DOE, Swedish Research Council.
Abstract (Originalsprache)
Reinforcement learning (RL) is a classical tool to solve network control or policy optimization problems in unknown environments. The original $Q$ -learning suffers from performance and complexity challenges across very large networks. Herein, a novel model-free ensemble reinforcement learning algorithm which adapts the classical $Q$ -learning is proposed to handle these challenges for networks which admit Markov decision process (MDP) models. Multiple $Q$ -learning algorithms are run on multiple, distinct, synthetically created and structurally related Markovian environments in parallel; the outputs are fused using an adaptive weighting mechanism based on the Jensen-Shannon divergence (JSD) to obtain an approximately optimal policy with low complexity. The theoretical justification of the algorithm, including the convergence of key statistics and $Q$ -functions are provided. Numerical results across several network models show that the proposed algorithm can achieve up to 55% less average policy error with up to 50% less runtime complexity than the state-of-the-art $Q$ -learning algorithms. Numerical results validate assumptions made in the theoretical analysis.
Förderung
- ARO
- DOE
- Swedish Research Council
- NSF
- ONR
- USC + Amazon Center on Secure and Trusted Machine Learning
Zitieren
APA
Bozkus, T., & Mitra, U. (2024). Multi-Timescale Ensemble $Q$-Learning for Markov Decision Process Policy Optimization. IEEE Transactions on Signal Processing, 72, 1427-1442. https://doi.org/10.1109/tsp.2024.3372699
Vancouver
Bozkus T, Mitra U. Multi-Timescale Ensemble $Q$-Learning for Markov Decision Process Policy Optimization. IEEE Transactions on Signal Processing. 2024;72:1427-1442. doi:10.1109/tsp.2024.3372699
BibTeX
@article{Bozkus2024,
author = {Bozkus, Talha and Mitra, Urbashi},
title = {Multi-Timescale Ensemble $Q$-Learning for Markov Decision Process Policy Optimization},
journal = {IEEE Transactions on Signal Processing},
year = {2024},
volume = {72},
pages = {1427--1442},
doi = {10.1109/tsp.2024.3372699}
}
EUVIMED