Attention With System Entropy for Optimizing Credit Assignment in Cooperative Multi-Agent Reinforcement Learning
IEEE Transactions on Automation Science and Engineering · 2025 · 22 · 14775-14787
DOI: 10.1109/tase.2025.3563725 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Automation Science and Engineering, 2025. Verfasst von Wei Wei und 4 weiteren Autor:innen. Die Arbeit wurde bisher 19-mal zitiert (Crossref) und nennt 41 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Beijing Natural Science Foundation, Fundamental Research Funds for the Central Universities.
Abstract (Originalsprache)
In cooperative multi-agent reinforcement learning (MARL), value function factorization methods have been proposed to address the dimensionality explosion problem encountered in centralized training and decentralized execution (CTDE). The existing value function factorization methods lack a perspective from the global system when addressing credit assignment, failing to measure each agent’s contribution to the current system. A typical limitation is that these factorization approaches primarily rely on the proximity of local individual groups to assign credit. There are challenges such as the sensitivity of credit assignment weights to local information and instability in the learning process. In this study, we focus on redistributing agents’ features during the system evolution and suggest employing an enhanced attention mechanism with system entropy measure to factorize value function. Specifically, this method emphasizes each agent’s representation between their local and global contributions and then redesigns multi-head attention to optimize the credit assignment in value function factorization. To evaluate the effectiveness of our method, we conduct a series of cooperative multi-agent tasks on the StarCraft II platform and compare the results with several state-of-the-art techniques, including Qatten, QMIX, QTRAN, COMA, and VDN. The experimental results demonstrate that our method achieves faster overall convergence speed, higher stability, and robust performance across various scenarios. Note to Practitioners—The fully cooperative multi-agent reinforcement learning (MARL) is extensively applied in practical multi-agent systems, such as coordinating multiple vehicle traffic conditions. Motivated by these practical application scenarios, our study is to enhance MARL’s learning efficiency by introducing an entropy-centered improved multi-head attention mechanism to address credit assignment issues. This method is aimed at measuring the importance weights of each agent within the agent system, thereby further boosting the learning efficiency of MARL. This work holds promise for application in large-scale intelligent agent systems, including multi-robot systems and collaborative multi-vehicle systems, among others.
Förderung
- National Natural Science Foundation of China
- Beijing Natural Science Foundation
- Fundamental Research Funds for the Central Universities
Zitieren
APA
Wei, W., Li, H., Zhou, S., Li, B., & Liu, X. (2025). Attention With System Entropy for Optimizing Credit Assignment in Cooperative Multi-Agent Reinforcement Learning. IEEE Transactions on Automation Science and Engineering, 22, 14775-14787. https://doi.org/10.1109/tase.2025.3563725
Vancouver
Wei W, Li H, Zhou S, Li B, Liu X. Attention With System Entropy for Optimizing Credit Assignment in Cooperative Multi-Agent Reinforcement Learning. IEEE Transactions on Automation Science and Engineering. 2025;22:14775-14787. doi:10.1109/tase.2025.3563725
BibTeX
@article{Wei2025,
author = {Wei, Wei and Li, Haibin and Zhou, Shiyuan and Li, Baifeng and Liu, Xue},
title = {Attention With System Entropy for Optimizing Credit Assignment in Cooperative Multi-Agent Reinforcement Learning},
journal = {IEEE Transactions on Automation Science and Engineering},
year = {2025},
volume = {22},
pages = {14775--14787},
doi = {10.1109/tase.2025.3563725}
}
EUVIMED