EUVIMEDEUVIMED entdecken
Crossref

Adaptively Calibrated Critic Estimates for Deep Reinforcement Learning

Nicolai Dorka, Tim Welschehold, Joschka Bodecker, Wolfram Burgard

IEEE Robotics and Automation Letters · 2023-02 · 8 (2) · 624-631

DOI: 10.1109/lra.2022.3229236 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Robotics and Automation Letters, 2023-02. Verfasst von Nicolai Dorka und 3 weiteren Autor:innen. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 40 Literaturangaben. Gefördert von: European Commission.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Accurate value estimates are important for off-policy reinforcement learning. Algorithms based on temporal difference learning typically are prone to an over- or underestimation bias building up over time. In this letter, we propose a general method called Adaptively Calibrated Critics (ACC) that uses the most recent high variance but unbiased on-policy rollouts to alleviate the bias of the low variance temporal difference targets. We apply ACC to Truncated Quantile Critics [1], which is an algorithm for continuous control that allows regulation of the bias with a hyperparameter tuned per environment. The resulting algorithm adaptively adjusts the parameter during training rendering hyperparameter search unnecessary and sets a new state of the art on the OpenAI gym continuous control benchmark among all algorithms that do not tune hyperparameters for each environment. ACC further achieves improved results on different tasks from the Meta-World robot benchmark. Additionally, we demonstrate the generality of ACC by applying it to TD3 [2] and showing an improved performance also in this setting.

Förderung

  • European Commission

Zitieren

APA

Dorka, N., Welschehold, T., Bodecker, J., & Burgard, W. (2023). Adaptively Calibrated Critic Estimates for Deep Reinforcement Learning. IEEE Robotics and Automation Letters, 8(2), 624-631. https://doi.org/10.1109/lra.2022.3229236

Vancouver

Dorka N, Welschehold T, Bodecker J, Burgard W. Adaptively Calibrated Critic Estimates for Deep Reinforcement Learning. IEEE Robotics and Automation Letters. 2023;8(2):624-631. doi:10.1109/lra.2022.3229236

BibTeX
@article{Dorka2023,
  author = {Dorka, Nicolai and Welschehold, Tim and Bodecker, Joschka and Burgard, Wolfram},
  title = {Adaptively Calibrated Critic Estimates for Deep Reinforcement Learning},
  journal = {IEEE Robotics and Automation Letters},
  year = {2023},
  volume = {8},
  number = {2},
  pages = {624--631},
  doi = {10.1109/lra.2022.3229236}
}
Kostenlos starten →