EUVIMEDEUVIMED entdecken
Crossref

Optimal Scheduling of Entropy Regularizer for Continuous-Time Linear-Quadratic Reinforcement Learning

Lukasz Szpruch, Tanut Treetanthiploet, Yufei Zhang

SIAM Journal on Control and Optimization · 29.02.2024 · 62 (1) · 135-166

DOI: 10.1137/22m1515744 ↗

Kurz gefasst

Fachartikel in der Zeitschrift SIAM Journal on Control and Optimization, online erschienen am 17.01.2024. Verfasst von Lukasz Szpruch und 2 weiteren Autor:innen. Die Arbeit wurde bisher 28-mal zitiert (Crossref) und nennt 36 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This work uses the entropy-regularized relaxed stochastic control perspective as a principled framework for designing reinforcement learning (RL) algorithms. Herein, an agent interacts with the environment by generating noisy controls distributed according to the optimal relaxed policy. The noisy policies, on the one hand, explore the space and hence facilitate learning, but, on the other hand, they introduce bias by assigning a positive probability to nonoptimal actions. This exploration-exploitation trade-off is determined by the strength of entropy regularization. We study algorithms resulting from two entropy regularization formulations: the exploratory control approach, where entropy is added to the cost objective, and the proximal policy update approach, where entropy penalizes policy divergence between consecutive episodes. We focus on the finite horizon continuous-time linear-quadratic (LQ) RL problem, where a linear dynamics with unknown drift coefficients is controlled subject to quadratic costs. In this setting, both algorithms yield a Gaussian relaxed policy. We quantify the precise difference between the value functions of a Gaussian policy and its noisy evaluation and show that the execution noise must be independent across time. By tuning the frequency of sampling from relaxed policies and the parameter governing the strength of entropy regularization, we prove that the regret, for both learning algorithms, is of the order [Formula: see text] (up to a logarithmic factor) over [Formula: see text] episodes, matching the best known result from the literature.

Zitieren

APA

Szpruch, L., Treetanthiploet, T., & Zhang, Y. (2024). Optimal Scheduling of Entropy Regularizer for Continuous-Time Linear-Quadratic Reinforcement Learning. SIAM Journal on Control and Optimization, 62(1), 135-166. https://doi.org/10.1137/22m1515744

Vancouver

Szpruch L, Treetanthiploet T, Zhang Y. Optimal Scheduling of Entropy Regularizer for Continuous-Time Linear-Quadratic Reinforcement Learning. SIAM Journal on Control and Optimization. 2024;62(1):135-166. doi:10.1137/22m1515744

BibTeX
@article{Szpruch2024,
  author = {Szpruch, Lukasz and Treetanthiploet, Tanut and Zhang, Yufei},
  title = {Optimal Scheduling of Entropy Regularizer for Continuous-Time Linear-Quadratic Reinforcement Learning},
  journal = {SIAM Journal on Control and Optimization},
  year = {2024},
  volume = {62},
  number = {1},
  pages = {135--166},
  doi = {10.1137/22m1515744}
}
Kostenlos starten →