EUVIMEDEUVIMED entdecken
Crossref

Offline Reinforcement Learning With Behavior Value Regularization

Longyang Huang, Botao Dong, Wei Xie, Weidong Zhang

IEEE Transactions on Cybernetics · 2024-06 · 54 (6) · 3692-3704

DOI: 10.1109/tcyb.2024.3385910 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Cybernetics, 2024-06. Verfasst von Longyang Huang und 3 weiteren Autor:innen. Die Arbeit wurde bisher 14-mal zitiert (Crossref) und nennt 42 Literaturangaben. Gefördert von: National Science and Technology Major Project, National Natural Science Foundation of China, Shanghai Science and Technology Program.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Offline reinforcement learning (offline RL) aims to find task-solving policies from prerecorded datasets without online environment interaction. It is unfortunate that extrapolation errors can cause over-optimistic Q-value estimates when learning with a fixed dataset, limiting the performance of the learned policy. To tackle this issue, this article proposes an offline actor-critic with behavior value regularization (OAC-BVR) method. In the policy evaluation stage, the difference between the Q-function and the value of the behavior policy is considered as the regularization term, driving the learned value function to approach the value of the behavior policy. The convergence of the proposed policy evaluation with behavior value regularization (PE-BVR) and the value function difference are analyzed, respectively. Compared with existing offline actor-critic methods, the proposed OAC-BVR method integrates the value of the behavior policy, thereby simultaneously alleviating over-optimistic Q-value estimates and reducing Q-function bias. Experimental results on the D4RL MuJoCo and Maze2d datasets demonstrate the validity of the proposed PE-BVR and the performance advantage of OAC-BVR over the state-of-the-art offline RL algorithms. The code of OAC-BVR is available at https://github.com/LongyangHuang/OAC-BVR.

Förderung

  • National Science and Technology Major Project
  • National Natural Science Foundation of China
  • Shanghai Science and Technology Program
  • Hainan Province Science and Technology Special Fund

Zitieren

APA

Huang, L., Dong, B., Xie, W., & Zhang, W. (2024). Offline Reinforcement Learning With Behavior Value Regularization. IEEE Transactions on Cybernetics, 54(6), 3692-3704. https://doi.org/10.1109/tcyb.2024.3385910

Vancouver

Huang L, Dong B, Xie W, Zhang W. Offline Reinforcement Learning With Behavior Value Regularization. IEEE Transactions on Cybernetics. 2024;54(6):3692-3704. doi:10.1109/tcyb.2024.3385910

BibTeX
@article{Huang2024,
  author = {Huang, Longyang and Dong, Botao and Xie, Wei and Zhang, Weidong},
  title = {Offline Reinforcement Learning With Behavior Value Regularization},
  journal = {IEEE Transactions on Cybernetics},
  year = {2024},
  volume = {54},
  number = {6},
  pages = {3692--3704},
  doi = {10.1109/tcyb.2024.3385910}
}
Kostenlos starten →