Efficient Offline Reinforcement Learning With Relaxed Conservatism
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-08 · 46 (8) · 5260-5272
DOI: 10.1109/tpami.2024.3364844 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-08. Verfasst von Longyang Huang und 2 weiteren Autor:innen. Die Arbeit wurde bisher 18-mal zitiert (Crossref) und nennt 44 Literaturangaben. Gefördert von: National Key R&D Program of China, Shanghai Science and Technology program, Hainan Province Science and Technology Special Fund.
Abstract (Originalsprache)
Offline reinforcement learning (RL) aims at learning an optimal policy from a static offline data set, without interacting with the environment. However, the theoretical understanding of the existing offline RL methods needs further studies, among which the conservatism of the learned Q-function and the learned policy is a major issue. In this article, we propose a simple and efficient offline RL with relaxed conservatism (ORL-RC) framework for addressing this concern by learning a Q-function that is close to the true Q-function under the learned policy. The conservatism of learned Q-functions and policies of offline RL methods is analyzed. The analysis results support that the conservatism can lead to policy performance degradation. We establish the convergence results of the proposed ORL-RC, and the bounds of learned Q-functions with and without sampling errors, respectively, suggesting that the gap between the learned Q-function and the true Q-function can be reduced by executing the conservative policy improvement. A practical implementation of ORL-RC is presented and the experimental results on the D4RL benchmark suggest that ORL-RC exhibits superior performance and substantially outperforms existing state-of-the-art offline RL methods.
Förderung
- National Key R&D Program of China
- Shanghai Science and Technology program
- Hainan Province Science and Technology Special Fund
- National Natural Science Foundation of China
Zitieren
APA
Huang, L., Dong, B., & Zhang, W. (2024). Efficient Offline Reinforcement Learning With Relaxed Conservatism. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(8), 5260-5272. https://doi.org/10.1109/tpami.2024.3364844
Vancouver
Huang L, Dong B, Zhang W. Efficient Offline Reinforcement Learning With Relaxed Conservatism. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(8):5260-5272. doi:10.1109/tpami.2024.3364844
BibTeX
@article{Huang2024,
author = {Huang, Longyang and Dong, Botao and Zhang, Weidong},
title = {Efficient Offline Reinforcement Learning With Relaxed Conservatism},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2024},
volume = {46},
number = {8},
pages = {5260--5272},
doi = {10.1109/tpami.2024.3364844}
}
EUVIMED