Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning
IEEE Robotics and Automation Letters · 2024-04 · 9 (4) · 3116-3123
DOI: 10.1109/lra.2024.3363530 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Robotics and Automation Letters, 2024-04. Verfasst von Suzan Ece Ada und 2 weiteren Autor:innen. Die Arbeit wurde bisher 20-mal zitiert (Crossref) und nennt 43 Literaturangaben. Gefördert von: JSPS KAKENHI, New Energy and Industrial Technology Development Organization, Türkiye Bilimsel ve Teknolojik Araştırma Kurumu.
Abstract (Originalsprache)
Offline Reinforcement Learning (RL) methods leverage previous experiences to learn better policies than the behavior policy used for data collection. However, they face challenges handling distribution shifts due to the lack of online interaction during training. To this end, we propose a novel method named State Reconstruction for Diffusion Policies (SRDP) that incorporates state reconstruction feature learning in the recent class of diffusion policies to address the problem of out-of-distribution (OOD) generalization. Our method promotes learning of generalizable state representation to alleviate the distribution shift caused by OOD states. To illustrate the OOD generalization and faster convergence of SRDP, we design a novel 2D Multimodal Contextual Bandit environment and realize it on a 6-DoF real-world UR10 robot, as well as in simulation, and compare its performance with prior algorithms. In particular, we show the importance of the proposed state reconstruction via ablation studies. In addition, we assess the performance of our model on standard continuous control benchmarks (D4RL), namely the navigation of an 8-DoF ant and forward locomotion of half-cheetah, hopper, and walker2d, achieving state-of-the-art results. Finally, we demonstrate that our method can achieve 167% improvement over the competing baseline on a sparse continuous control navigation task where various regions of the state space are removed from the offline RL dataset, including the region encapsulating the goal.
Förderung
- JSPS KAKENHI
- New Energy and Industrial Technology Development Organization
- Türkiye Bilimsel ve Teknolojik Araştırma Kurumu
- European Commission
- INVERSE Project
Zitieren
APA
Ada, S. E., Oztop, E., & Ugur, E. (2024). Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning. IEEE Robotics and Automation Letters, 9(4), 3116-3123. https://doi.org/10.1109/lra.2024.3363530
Vancouver
Ada SE, Oztop E, Ugur E. Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning. IEEE Robotics and Automation Letters. 2024;9(4):3116-3123. doi:10.1109/lra.2024.3363530
BibTeX
@article{Ada2024,
author = {Ada, Suzan Ece and Oztop, Erhan and Ugur, Emre},
title = {Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning},
journal = {IEEE Robotics and Automation Letters},
year = {2024},
volume = {9},
number = {4},
pages = {3116--3123},
doi = {10.1109/lra.2024.3363530}
}
EUVIMED