Using Reward Machines for Offline Reinforcement Learning With Non-Markovian Reward Functions
Scientific Insights · 2025
DOI: 10.64897/si.2025v1i1.005 ↗Kurz gefasst
Fachartikel in der Zeitschrift Scientific Insights, 2025. Verfasst von Yanze Wang. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
We investigate the problem of offline reinforcement learning using non-Markovian reward functions, which allows for the incorporation of more realistic and intricate reward structures in the learning process. Offline reinforcement learning has shown promising potential in learning optimal policies when the agent has access to previously collected static datasets. Reward machines offer a way to encode the high-level structure of non-Markovian reward functions. We introduce C-QRM, an offline reinforcement learning approach that employs non-Markovian reward functions specified as reward machines to accomplish complex tasks and learn an optimal policy more efficiently by utilizing the offline dataset. Our objective is to learn a conservative Q-function that decomposes complex high-level reward machines and whose expected value of a policy under this Q-function provides a lower bound to its actual value. C-QRM learns these lower-bounded Q-values, mitigating overestimation bias and improving sampling efficiency. We evaluate the performance of the proposed C-QRM algorithm by comparing it to QRM as a baseline method. The results indicate that C-QRM outperforms QRM with fewer training steps and benefits from the offline dataset.
Zitieren
APA
Wang, Y. (2025). Using Reward Machines for Offline Reinforcement Learning With Non-Markovian Reward Functions. Scientific Insights. https://doi.org/10.64897/si.2025v1i1.005
Vancouver
Wang Y. Using Reward Machines for Offline Reinforcement Learning With Non-Markovian Reward Functions. Scientific Insights. 2025. doi:10.64897/si.2025v1i1.005
BibTeX
@article{Wang2025,
author = {Wang, Yanze},
title = {Using Reward Machines for Offline Reinforcement Learning With Non-Markovian Reward Functions},
journal = {Scientific Insights},
year = {2025},
doi = {10.64897/si.2025v1i1.005}
}
EUVIMED