EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

An Overestimation Reduction Method Based on the Multi-step Weighted Double Estimation Using Value-Decomposition Multi-agent Reinforcement Learning

Li-yang Zhao, Tian-qing Chang, Li-bin Guo, Jie Zhang, Lei Zhang, Jin-dun Ma

Neural Processing Letters · 23.04.2024 · 56 (3)

DOI: 10.1007/s11063-024-11611-2 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Neural Processing Letters, online erschienen am 23.04.2024. Verfasst von Li-yang Zhao und 5 weiteren Autor:innen. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 49 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The joint action-value function (JAVF) plays a key role in the centralized training of multi-agent deep reinforcement learning (MADRL)-based algorithms using the value function decomposition (VFD) and in the generating process of a collaborative policy between agents. However, under the influence of multiple factors such as environmental noise, inadequate exploration and iterative updating mechanism, estimation bias is inevitably introduced, causing its overestimation problem, which in turn prevents agents from obtaining accurate reward signals during the learning process, and fails to correctly approximate the optimal policy. To address this problem, this paper first analyzes the causes of joint action-value function overestimation, gives the corresponding mathematical proofs and theoretical derivations, and obtains the lower bound of the overestimation error; then, a MADRL overestimation reduction method based on the multi-step weighted double estimation named λWD QMIX is proposed. Specifically, the λWD QMIX method effectively achieves more stable and accurate JAVF estimation results using the bias correction estimation mechanisms based on the weighted double estimation and multi-step updating based on eligibility trace backup, without additionally adding or changing any network structure. The results of a series of experiments on the StarCraft II micromanipulation benchmark show that the proposed λWD QMIX algorithm can effectively improve the final performance and learning efficiency of the baseline algorithm, and can be seamlessly integrated with the partially MADRL algorithms based on communication learning.

Zitieren

APA

Zhao, L., Chang, T., Guo, L., Zhang, J., Zhang, L., & Ma, J. (2024). An Overestimation Reduction Method Based on the Multi-step Weighted Double Estimation Using Value-Decomposition Multi-agent Reinforcement Learning. Neural Processing Letters, 56(3). https://doi.org/10.1007/s11063-024-11611-2

Vancouver

Zhao L, Chang T, Guo L, Zhang J, Zhang L, Ma J. An Overestimation Reduction Method Based on the Multi-step Weighted Double Estimation Using Value-Decomposition Multi-agent Reinforcement Learning. Neural Processing Letters. 2024;56(3). doi:10.1007/s11063-024-11611-2

BibTeX
@article{Zhao2024,
  author = {Zhao, Li-yang and Chang, Tian-qing and Guo, Li-bin and Zhang, Jie and Zhang, Lei and Ma, Jin-dun},
  title = {An Overestimation Reduction Method Based on the Multi-step Weighted Double Estimation Using Value-Decomposition Multi-agent Reinforcement Learning},
  journal = {Neural Processing Letters},
  year = {2024},
  volume = {56},
  number = {3},
  doi = {10.1007/s11063-024-11611-2}
}
Kostenlos starten →