EUVIMEDEUVIMED entdecken
Crossref

Efficient Offline Reinforcement Learning With Relaxed Conservatism

Longyang Huang, Botao Dong, Weidong Zhang

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-08 · 46 (8) · 5260-5272

DOI: 10.1109/tpami.2024.3364844 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-08. Verfasst von Longyang Huang und 2 weiteren Autor:innen. Die Arbeit wurde bisher 18-mal zitiert (Crossref) und nennt 44 Literaturangaben. Gefördert von: National Key R&D Program of China, Shanghai Science and Technology program, Hainan Province Science and Technology Special Fund.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Offline reinforcement learning (RL) aims at learning an optimal policy from a static offline data set, without interacting with the environment. However, the theoretical understanding of the existing offline RL methods needs further studies, among which the conservatism of the learned Q-function and the learned policy is a major issue. In this article, we propose a simple and efficient offline RL with relaxed conservatism (ORL-RC) framework for addressing this concern by learning a Q-function that is close to the true Q-function under the learned policy. The conservatism of learned Q-functions and policies of offline RL methods is analyzed. The analysis results support that the conservatism can lead to policy performance degradation. We establish the convergence results of the proposed ORL-RC, and the bounds of learned Q-functions with and without sampling errors, respectively, suggesting that the gap between the learned Q-function and the true Q-function can be reduced by executing the conservative policy improvement. A practical implementation of ORL-RC is presented and the experimental results on the D4RL benchmark suggest that ORL-RC exhibits superior performance and substantially outperforms existing state-of-the-art offline RL methods.

Förderung

  • National Key R&D Program of China
  • Shanghai Science and Technology program
  • Hainan Province Science and Technology Special Fund
  • National Natural Science Foundation of China

Zitieren

APA

Huang, L., Dong, B., & Zhang, W. (2024). Efficient Offline Reinforcement Learning With Relaxed Conservatism. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(8), 5260-5272. https://doi.org/10.1109/tpami.2024.3364844

Vancouver

Huang L, Dong B, Zhang W. Efficient Offline Reinforcement Learning With Relaxed Conservatism. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(8):5260-5272. doi:10.1109/tpami.2024.3364844

BibTeX
@article{Huang2024,
  author = {Huang, Longyang and Dong, Botao and Zhang, Weidong},
  title = {Efficient Offline Reinforcement Learning With Relaxed Conservatism},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2024},
  volume = {46},
  number = {8},
  pages = {5260--5272},
  doi = {10.1109/tpami.2024.3364844}
}
Kostenlos starten →