A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (15) · 16908-16916
DOI: 10.1609/aaai.v38i15.29633 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Yinmin Zhang und 6 weiteren Autor:innen. Die Arbeit wurde bisher 20-mal zitiert (Crossref).
Abstract (Originalsprache)
Offline-to-online Reinforcement Learning (O2O RL) aims to improve the performance of offline pretrained policy using only a few online samples. Built on offline RL algorithms, most O2O methods focus on the balance between RL objective and pessimism, or the utilization of offline and online samples. In this paper, from a novel perspective, we systematically study the challenges that remain in O2O RL and identify that the reason behind the slow improvement of the performance and the instability of online finetuning lies in the inaccurate Q-value estimation inherited from offline pretraining. Specifically, we demonstrate that the estimation bias and the inaccurate rank of Q-value cause a misleading signal for the policy update, making the standard offline RL algorithms, such as CQL and TD3-BC, ineffective in the online finetuning. Based on this observation, we address the problem of Q-value estimation by two techniques: (1) perturbed value update and (2) increased frequency of Q-value updates. The first technique smooths out biased Q-value estimation with sharp peaks, preventing early-stage policy exploitation of sub-optimal actions. The second one alleviates the estimation bias inherited from offline pretraining by accelerating learning. Extensive experiments on the MuJoco and Adroit environments demonstrate that the proposed method, named SO2, significantly alleviates Q-value estimation issues, and consistently improves the performance against the state-of-the-art methods by up to 83.1%.
Zitieren
APA
Zhang, Y., Liu, J., Li, C., Niu, Y., Yang, Y., Liu, Y., & Ouyang, W. (2024). A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 38(15), 16908-16916. https://doi.org/10.1609/aaai.v38i15.29633
Vancouver
Zhang Y, Liu J, Li C, Niu Y, Yang Y, Liu Y, et al. A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(15):16908-16916. doi:10.1609/aaai.v38i15.29633
BibTeX
@article{Zhang2024,
author = {Zhang, Yinmin and Liu, Jie and Li, Chuming and Niu, Yazhe and Yang, Yaodong and Liu, Yu and Ouyang, Wanli},
title = {A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2024},
volume = {38},
number = {15},
pages = {16908--16916},
doi = {10.1609/aaai.v38i15.29633}
}
EUVIMED