Augmented Proximal Policy Optimization for Safe Reinforcement Learning
Proceedings of the AAAI Conference on Artificial Intelligence · 26.06.2023 · 37 (6) · 7288-7295
DOI: 10.1609/aaai.v37i6.25888 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 26.06.2023. Verfasst von Juntao Dai und 4 weiteren Autor:innen. Die Arbeit wurde bisher 18-mal zitiert (Crossref).
Abstract (Originalsprache)
Safe reinforcement learning considers practical scenarios that maximize the return while satisfying safety constraints. Current algorithms, which suffer from training oscillations or approximation errors, still struggle to update the policy efficiently with precise constraint satisfaction. In this article, we propose Augmented Proximal Policy Optimization (APPO), which augments the Lagrangian function of the primal constrained problem via attaching a quadratic deviation term. The constructed multiplier-penalty function dampens cost oscillation for stable convergence while being equivalent to the primal constrained problem to precisely control safety costs. APPO alternately updates the policy and the Lagrangian multiplier via solving the constructed augmented primal-dual problem, which can be easily implemented by any first-order optimizer. We apply our APPO methods in diverse safety-constrained tasks, setting a new state of the art compared with a comprehensive list of safe RL baselines. Extensive experiments verify the merits of our method in easy implementation, stable convergence, and precise cost control.
Zitieren
APA
Dai, J., Ji, J., Yang, L., Zheng, Q., & Pan, G. (2023). Augmented Proximal Policy Optimization for Safe Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 37(6), 7288-7295. https://doi.org/10.1609/aaai.v37i6.25888
Vancouver
Dai J, Ji J, Yang L, Zheng Q, Pan G. Augmented Proximal Policy Optimization for Safe Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence. 2023;37(6):7288-7295. doi:10.1609/aaai.v37i6.25888
BibTeX
@article{Dai2023,
author = {Dai, Juntao and Ji, Jiaming and Yang, Long and Zheng, Qian and Pan, Gang},
title = {Augmented Proximal Policy Optimization for Safe Reinforcement Learning},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2023},
volume = {37},
number = {6},
pages = {7288--7295},
doi = {10.1609/aaai.v37i6.25888}
}
EUVIMED