Traffic expertise meets residual RL: Knowledge-informed model-based residual reinforcement learning for CAV trajectory control
Communications in Transportation Research · 2024-12 · 4 · 100142
DOI: 10.1016/j.commtr.2024.100142 ↗Kurz gefasst
Fachartikel in der Zeitschrift Communications in Transportation Research, 2024-12. Verfasst von Zihao Sheng und 2 weiteren Autor:innen. Die Arbeit wurde bisher 36-mal zitiert (Crossref) und nennt 78 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: US Department of Transportation, University Transportation Center, Missouri University of Science and Technology.
Abstract (Originalsprache)
Model-based reinforcement learning (RL) is anticipated to exhibit higher sample efficiency compared to model-free RL by utilizing a virtual environment model. However, it is challenging to obtain sufficiently accurate representations of the environmental dynamics due to uncertainties in complex systems and environments. An inaccurate environment model may degrade the sample efficiency and performance of model-based RL. Furthermore, while model-based RL can improve sample efficiency, it often still requires substantial training time to learn from scratch, potentially limiting its advantages over model-free approaches. To address these challenges, this paper introduces a knowledge-informed model-based residual reinforcement learning framework aimed at enhancing learning efficiency by infusing established expert knowledge into the learning process and avoiding the issue of beginning from zero. Our approach integrates traffic expert knowledge into a virtual environment model, employing the Intelligent Driver Model (IDM) for basic dynamics and neural networks for residual dynamics, thus ensuring adaptability to complex scenarios. We propose a novel strategy that combines traditional control methods with residual RL, facilitating efficient learning and policy optimization without the need to learn from scratch. The proposed approach is applied to CAV trajectory control tasks for the dissipation of stop-and-go waves in mixed traffic flow. Experimental results demonstrate that our proposed approach enables the CAV agent to achieve superior performance in trajectory control compared to the baseline agents in terms of sample efficiency, traffic flow smoothness and traffic mobility. The source code and supplementary materials are available at: https://zihaosheng.github.io/traffic-expertise-RL/.
Förderung
- US Department of Transportation
- University Transportation Center, Missouri University of Science and Technology
Zitieren
APA
Sheng, Z., Huang, Z., & Chen, S. (2024). Traffic expertise meets residual RL: Knowledge-informed model-based residual reinforcement learning for CAV trajectory control. Communications in Transportation Research, 4, 100142. https://doi.org/10.1016/j.commtr.2024.100142
Vancouver
Sheng Z, Huang Z, Chen S. Traffic expertise meets residual RL: Knowledge-informed model-based residual reinforcement learning for CAV trajectory control. Communications in Transportation Research. 2024;4:100142. doi:10.1016/j.commtr.2024.100142
BibTeX
@article{Sheng2024,
author = {Sheng, Zihao and Huang, Zilin and Chen, Sikai},
title = {Traffic expertise meets residual RL: Knowledge-informed model-based residual reinforcement learning for CAV trajectory control},
journal = {Communications in Transportation Research},
year = {2024},
volume = {4},
pages = {100142},
doi = {10.1016/j.commtr.2024.100142}
}
EUVIMED