EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Concurrent Learning of Control Policy and Unknown Safety Specifications in Reinforcement Learning

Lunet Yifru, Ali Baheri

IEEE Open Journal of Control Systems · 2024 · 3 · 266-281

DOI: 10.1109/ojcsys.2024.3418306 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Open Journal of Control Systems, 2024. Verfasst von Lunet Yifru und 1 weiteren Autor:innen. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 73 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Reinforcement learning (RL) has revolutionized decision-making across a wide range of domains over the past few decades. Yet, deploying RL policies in real-world scenarios presents the crucial challenge of ensuring safety. Traditional safe RL approaches have predominantly focused on incorporating predefined safety constraints into the policy learning process. However, this reliance on predefined safety constraints poses limitations in dynamic and unpredictable real-world settings where such constraints may not be available or sufficiently adaptable. Bridging this gap, we propose a novel approach that concurrently learns a safe RL control policy and identifies the unknown safety constraint parameters of a given environment. Initializing with a parametric signal temporal logic (pSTL) safety specification and a small initial labeled dataset, we frame the problem as a bilevel optimization task, intricately integrating constrained policy optimization, using a Lagrangian-variant of the twin delayed deep deterministic policy gradient (TD3) algorithm, with Bayesian optimization for optimizing parameters for the given pSTL safety specification. Through experimentation in comprehensive case studies, we validate the efficacy of this approach across varying forms of environmental constraints, consistently yielding safe RL policies with high returns. Furthermore, our findings indicate successful learning of STL safety constraint parameters, exhibiting a high degree of conformity with true environmental safety constraints. The performance of our model closely mirrors that of an ideal scenario that possesses complete prior knowledge of safety constraints, demonstrating its proficiency in accurately identifying environmental safety constraints and learning safe policies that adhere to those constraints. A Python implementation of the algorithm can be found at https://github.com/SAILRIT/Concurrent-Learning-of-Control-Policy-and-Unknown-Constraints-in-Reinforcement-Learning.git.

Zitieren

APA

Yifru, L., & Baheri, A. (2024). Concurrent Learning of Control Policy and Unknown Safety Specifications in Reinforcement Learning. IEEE Open Journal of Control Systems, 3, 266-281. https://doi.org/10.1109/ojcsys.2024.3418306

Vancouver

Yifru L, Baheri A. Concurrent Learning of Control Policy and Unknown Safety Specifications in Reinforcement Learning. IEEE Open Journal of Control Systems. 2024;3:266-281. doi:10.1109/ojcsys.2024.3418306

BibTeX
@article{Yifru2024,
  author = {Yifru, Lunet and Baheri, Ali},
  title = {Concurrent Learning of Control Policy and Unknown Safety Specifications in Reinforcement Learning},
  journal = {IEEE Open Journal of Control Systems},
  year = {2024},
  volume = {3},
  pages = {266--281},
  doi = {10.1109/ojcsys.2024.3418306}
}
Kostenlos starten →