EUVIMEDEUVIMED entdecken
Crossref

A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning

Yinmin Zhang, Jie Liu, Chuming Li, Yazhe Niu, Yaodong Yang, Yu Liu, Wanli Ouyang

Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (15) · 16908-16916

DOI: 10.1609/aaai.v38i15.29633 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Yinmin Zhang und 6 weiteren Autor:innen. Die Arbeit wurde bisher 20-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Offline-to-online Reinforcement Learning (O2O RL) aims to improve the performance of offline pretrained policy using only a few online samples. Built on offline RL algorithms, most O2O methods focus on the balance between RL objective and pessimism, or the utilization of offline and online samples. In this paper, from a novel perspective, we systematically study the challenges that remain in O2O RL and identify that the reason behind the slow improvement of the performance and the instability of online finetuning lies in the inaccurate Q-value estimation inherited from offline pretraining. Specifically, we demonstrate that the estimation bias and the inaccurate rank of Q-value cause a misleading signal for the policy update, making the standard offline RL algorithms, such as CQL and TD3-BC, ineffective in the online finetuning. Based on this observation, we address the problem of Q-value estimation by two techniques: (1) perturbed value update and (2) increased frequency of Q-value updates. The first technique smooths out biased Q-value estimation with sharp peaks, preventing early-stage policy exploitation of sub-optimal actions. The second one alleviates the estimation bias inherited from offline pretraining by accelerating learning. Extensive experiments on the MuJoco and Adroit environments demonstrate that the proposed method, named SO2, significantly alleviates Q-value estimation issues, and consistently improves the performance against the state-of-the-art methods by up to 83.1%.

Zitieren

APA

Zhang, Y., Liu, J., Li, C., Niu, Y., Yang, Y., Liu, Y., & Ouyang, W. (2024). A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 38(15), 16908-16916. https://doi.org/10.1609/aaai.v38i15.29633

Vancouver

Zhang Y, Liu J, Li C, Niu Y, Yang Y, Liu Y, et al. A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(15):16908-16916. doi:10.1609/aaai.v38i15.29633

BibTeX
@article{Zhang2024,
  author = {Zhang, Yinmin and Liu, Jie and Li, Chuming and Niu, Yazhe and Yang, Yaodong and Liu, Yu and Ouyang, Wanli},
  title = {A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2024},
  volume = {38},
  number = {15},
  pages = {16908--16916},
  doi = {10.1609/aaai.v38i15.29633}
}
Kostenlos starten →