HH-MAPPO: A Hierarchical Reinforcement Learning Framework for Dynamic-Scale Target–Attacker–Defender Games
Entropy · 13.07.2026 · 28 (7) · 793
DOI: 10.3390/e28070793 ↗Kurz gefasst
Fachartikel in der Zeitschrift Entropy, online erschienen am 13.07.2026. Verfasst von Junhui Huang und 6 weiteren Autor:innen. Beteiligte Einrichtung: School of Communication Engineering, Army Engineering University of PLA, Nanjing 210000, China. Die Arbeit nennt 40 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, Natural Science Foundation of Jiangsu Province.
Abstract (Originalsprache)
The Target–Attacker–Defender (TAD) pursuit–evasion game is a core challenge in multi-agent cooperative control, yet real-world settings involving dynamic team scaling and strict energy constraints remain largely unaddressed. When scalable shared-parameter policies are adopted to cope with the varying number of agents, severe policy homogeneity emerges, preventing effective division of labor. This paper proposes a Hierarchical Heterogeneous Multi-Agent Proximal Policy Optimization (HH-MAPPO) framework to resolve these challenges. Both levels employ actor–critic networks with Role-Aware Embedding (RAE). In this mechanism, each agent is assigned a unique, learnable role embedding derived from its identity. These embeddings serve as conditioning inputs to the shared policy network, enabling it to generate differentiated behaviors and effectively mitigating policy homogeneity. The upper-level policy determines the number of defenders to deploy and assigns interception targets, while the lower-level policy handles continuous control of each defender and the ground moving target (GMT). This hierarchy resolves dynamic observation spaces via a target-matching mechanism, where each defender’s observation includes only its own state and its assigned attacker’s state, keeping observation dimension constant. Experiments in a 3D TAD simulation with continuous attacker arrivals and energy-constrained defenders show the following: (1) HH-MAPPO achieves superior interception performance compared to baseline methods in both symmetric and asymmetric scenarios; (2) ablation studies confirm RAE increases policy diversity, raising Sequence-Based Action Dissimilarity (SBAD) by 15.5%; and (3) Pareto analysis demonstrates a superior performance–energy trade-off, maintaining about 70% interception rate even under an extreme energy cap (E = 30).
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- School of Communication Engineering, Army Engineering University of PLA, Nanjing 210000, China.
- School of Command and Control Engineering, Army Engineering University of PLA, Nanjing 210000, China.
ORCID-Profile: Junhui Huang
Förderung
- National Natural Science Foundation of China
- Natural Science Foundation of Jiangsu Province
Zitieren
APA
Huang, J., Guo, Y., Chen, X., Wei, J., Yi, J., Chen, X., & Chen, L. (2026). HH-MAPPO: A Hierarchical Reinforcement Learning Framework for Dynamic-Scale Target–Attacker–Defender Games. Entropy, 28(7), 793. https://doi.org/10.3390/e28070793
Vancouver
Huang J, Guo Y, Chen X, Wei J, Yi J, Chen X, et al. HH-MAPPO: A Hierarchical Reinforcement Learning Framework for Dynamic-Scale Target–Attacker–Defender Games. Entropy. 2026;28(7):793. doi:10.3390/e28070793
BibTeX
@article{Huang2026,
author = {Huang, Junhui and Guo, Yan and Chen, Xiliang and Wei, Jianyu and Yi, Jiawei and Chen, Xinliang and Chen, Lifeng},
title = {HH-MAPPO: A Hierarchical Reinforcement Learning Framework for Dynamic-Scale Target–Attacker–Defender Games},
journal = {Entropy},
year = {2026},
volume = {28},
number = {7},
pages = {793},
doi = {10.3390/e28070793}
}
EUVIMED