EUVIMEDEUVIMED entdecken
Crossref

Variance Reduced Domain Randomization for Reinforcement Learning With Policy Gradient

Yuankun Jiang, Chenglin Li, Wenrui Dai, Junni Zou, Hongkai Xiong

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-02 · 46 (2) · 1031-1048

DOI: 10.1109/tpami.2023.3330332 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-02. Verfasst von Yuankun Jiang und 4 weiteren Autor:innen. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 51 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Program of Shanghai Science and Technology Innovation, Key Technology Research and Development Program of Shandong Province.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

By introducing randomness on the environments, domain randomization (DR) imposes diversity to the policy training of deep reinforcement learning, and thus improves its capability of generalization. The randomization of environments, however, introduces another source of variability for the estimate of policy gradients, in addition to the already high variance incurred by trajectory sampling. Therefore, with standard state-dependent baselines, the policy gradient methods may still suffer high variance, causing a low sample efficiency during the training of DR. In this paper, we theoretically derive a bias-free and state/environment-dependent optimal baseline for DR, and analytically show its ability to achieve further variance reduction over the standard constant and state-dependent baselines for DR. Based on our theory, we further propose a variance reduced domain randomization (VRDR) approach for policy gradient methods, to strike a tradeoff between the variance reduction and computational complexity for the practical implementation. By dividing the entire space of environments into some subspaces and then estimating the state/subspace-dependent baseline, VRDR enjoys a theoretical guarantee of variance reduction and faster convergence than the state-dependent baselines. Empirical evaluations on six robot control tasks with randomized dynamics demonstrate that VRDR not only accelerates the convergence of policy training, but can consistently achieve a better eventual policy with improved training stability.

Förderung

  • National Natural Science Foundation of China
  • Program of Shanghai Science and Technology Innovation
  • Key Technology Research and Development Program of Shandong Province

Zitieren

APA

Jiang, Y., Li, C., Dai, W., Zou, J., & Xiong, H. (2024). Variance Reduced Domain Randomization for Reinforcement Learning With Policy Gradient. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(2), 1031-1048. https://doi.org/10.1109/tpami.2023.3330332

Vancouver

Jiang Y, Li C, Dai W, Zou J, Xiong H. Variance Reduced Domain Randomization for Reinforcement Learning With Policy Gradient. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(2):1031-1048. doi:10.1109/tpami.2023.3330332

BibTeX
@article{Jiang2024,
  author = {Jiang, Yuankun and Li, Chenglin and Dai, Wenrui and Zou, Junni and Xiong, Hongkai},
  title = {Variance Reduced Domain Randomization for Reinforcement Learning With Policy Gradient},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2024},
  volume = {46},
  number = {2},
  pages = {1031--1048},
  doi = {10.1109/tpami.2023.3330332}
}
Kostenlos starten →