EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Exploring the Use of Invalid Action Masking in Reinforcement Learning: A Comparative Study of On-Policy and Off-Policy Algorithms in Real-Time Strategy Games

Yueqi Hou, Xiaolong Liang, Jiaqiang Zhang, Qisong Yang, Aiwu Yang, Ning Wang

Applied Sciences · 18.07.2023 · 13 (14) · 8283

DOI: 10.3390/app13148283 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 18.07.2023. Verfasst von Yueqi Hou und 5 weiteren Autor:innen. Die Arbeit wurde bisher 38-mal zitiert (Crossref) und nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Invalid action masking is a practical technique in deep reinforcement learning to prevent agents from taking invalid actions. Existing approaches rely on action masking during policy training and utilization. This study focuses on developing reinforcement learning algorithms that incorporate action masking during training but can be used without action masking during policy execution. The study begins by conducting a theoretical analysis to elucidate the distinction between naive policy gradient and invalid action policy gradient. Based on this analysis, we demonstrate that the naive policy gradient is a valid gradient and is equivalent to the proposed composite objective algorithm, which optimizes both the masked policy and the original policy in parallel. Moreover, we propose an off-policy algorithm for invalid action masking that employs the masked policy for sampling while optimizing the original policy. To compare the effectiveness of these algorithms, experiments are conducted using a simplified real-time strategy (RTS) game simulator called Gym-μRTS. Based on empirical findings, we recommend utilizing the off-policy algorithm for addressing most tasks while employing the composite objective algorithm for handling more complex tasks.

Förderung

  • National Natural Science Foundation of China

Zitieren

APA

Hou, Y., Liang, X., Zhang, J., Yang, Q., Yang, A., & Wang, N. (2023). Exploring the Use of Invalid Action Masking in Reinforcement Learning: A Comparative Study of On-Policy and Off-Policy Algorithms in Real-Time Strategy Games. Applied Sciences, 13(14), 8283. https://doi.org/10.3390/app13148283

Vancouver

Hou Y, Liang X, Zhang J, Yang Q, Yang A, Wang N. Exploring the Use of Invalid Action Masking in Reinforcement Learning: A Comparative Study of On-Policy and Off-Policy Algorithms in Real-Time Strategy Games. Applied Sciences. 2023;13(14):8283. doi:10.3390/app13148283

BibTeX
@article{Hou2023,
  author = {Hou, Yueqi and Liang, Xiaolong and Zhang, Jiaqiang and Yang, Qisong and Yang, Aiwu and Wang, Ning},
  title = {Exploring the Use of Invalid Action Masking in Reinforcement Learning: A Comparative Study of On-Policy and Off-Policy Algorithms in Real-Time Strategy Games},
  journal = {Applied Sciences},
  year = {2023},
  volume = {13},
  number = {14},
  pages = {8283},
  doi = {10.3390/app13148283}
}
Kostenlos starten →