EUVIMEDEUVIMED entdecken
Crossref

Adjacency Constraint for Efficient Hierarchical Reinforcement Learning

Tianren Zhang, Shangqi Guo, Tian Tan, Xiaolin Hu, Feng Chen

IEEE Transactions on Pattern Analysis and Machine Intelligence · 01.04.2023 · 45 (4) · 4152-4166

DOI: 10.1109/tpami.2022.3192418 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, erschienen am 01.04.2023. Verfasst von Tianren Zhang und 4 weiteren Autor:innen. Die Arbeit wurde bisher 34-mal zitiert (Crossref) und nennt 59 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Tsinghua-Guoqiang Research Program, National Key Research and Development Program of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Goal-conditioned Hierarchical Reinforcement Learning (HRL) is a promising approach for scaling up reinforcement learning (RL) techniques. However, it often suffers from training inefficiency as the action space of the high-level, i.e., the goal space, is large. Searching in a large goal space poses difficulty for both high-level subgoal generation and low-level policy learning. In this article, we show that this problem can be effectively alleviated by restricting the high-level action space from the whole goal space to a k-step adjacent region of the current state using an adjacency constraint. We theoretically prove that in a deterministic Markov Decision Process (MDP), the proposed adjacency constraint preserves the optimal hierarchical policy, while in a stochastic MDP the adjacency constraint induces a bounded state-value suboptimality determined by the MDP's transition structure. We further show that this constraint can be practically implemented by training an adjacency network that can discriminate between adjacent and non-adjacent subgoals. Experimental results on discrete and continuous control tasks including challenging simulated robot locomotion and manipulation tasks show that incorporating the adjacency constraint significantly boosts the performance of state-of-the-art goal-conditioned HRL approaches.

Förderung

  • National Natural Science Foundation of China
  • Tsinghua-Guoqiang Research Program
  • National Key Research and Development Program of China

Zitieren

APA

Zhang, T., Guo, S., Tan, T., Hu, X., & Chen, F. (2023). Adjacency Constraint for Efficient Hierarchical Reinforcement Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4), 4152-4166. https://doi.org/10.1109/tpami.2022.3192418

Vancouver

Zhang T, Guo S, Tan T, Hu X, Chen F. Adjacency Constraint for Efficient Hierarchical Reinforcement Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2023;45(4):4152-4166. doi:10.1109/tpami.2022.3192418

BibTeX
@article{Zhang2023,
  author = {Zhang, Tianren and Guo, Shangqi and Tan, Tian and Hu, Xiaolin and Chen, Feng},
  title = {Adjacency Constraint for Efficient Hierarchical Reinforcement Learning},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2023},
  volume = {45},
  number = {4},
  pages = {4152--4166},
  doi = {10.1109/tpami.2022.3192418}
}
Kostenlos starten →