EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

HH-MAPPO: A Hierarchical Reinforcement Learning Framework for Dynamic-Scale Target–Attacker–Defender Games

Junhui Huang iD, Yan Guo, Xiliang Chen, Jianyu Wei, Jiawei Yi, Xinliang Chen, Lifeng Chen

Entropy · 13.07.2026 · 28 (7) · 793

DOI: 10.3390/e28070793 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Entropy, online erschienen am 13.07.2026. Verfasst von Junhui Huang und 6 weiteren Autor:innen. Beteiligte Einrichtung: School of Communication Engineering, Army Engineering University of PLA, Nanjing 210000, China. Die Arbeit nennt 40 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, Natural Science Foundation of Jiangsu Province.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The Target–Attacker–Defender (TAD) pursuit–evasion game is a core challenge in multi-agent cooperative control, yet real-world settings involving dynamic team scaling and strict energy constraints remain largely unaddressed. When scalable shared-parameter policies are adopted to cope with the varying number of agents, severe policy homogeneity emerges, preventing effective division of labor. This paper proposes a Hierarchical Heterogeneous Multi-Agent Proximal Policy Optimization (HH-MAPPO) framework to resolve these challenges. Both levels employ actor–critic networks with Role-Aware Embedding (RAE). In this mechanism, each agent is assigned a unique, learnable role embedding derived from its identity. These embeddings serve as conditioning inputs to the shared policy network, enabling it to generate differentiated behaviors and effectively mitigating policy homogeneity. The upper-level policy determines the number of defenders to deploy and assigns interception targets, while the lower-level policy handles continuous control of each defender and the ground moving target (GMT). This hierarchy resolves dynamic observation spaces via a target-matching mechanism, where each defender’s observation includes only its own state and its assigned attacker’s state, keeping observation dimension constant. Experiments in a 3D TAD simulation with continuous attacker arrivals and energy-constrained defenders show the following: (1) HH-MAPPO achieves superior interception performance compared to baseline methods in both symmetric and asymmetric scenarios; (2) ablation studies confirm RAE increases policy diversity, raising Sequence-Based Action Dissimilarity (SBAD) by 15.5%; and (3) Pareto analysis demonstrates a superior performance–energy trade-off, maintaining about 70% interception rate even under an extreme energy cap (E = 30).

Schlagwörter

Stichwörter der Autor:innen

behavioral diversityenergy efficiencyhierarchical reinforcement learningmulti-agentrole-aware embeddingsystemtarget–attacker–defender game

Autor:innen und Institutionen

  • School of Communication Engineering, Army Engineering University of PLA, Nanjing 210000, China.
  • School of Command and Control Engineering, Army Engineering University of PLA, Nanjing 210000, China.

ORCID-Profile: Junhui Huang

Förderung

  • National Natural Science Foundation of China
  • Natural Science Foundation of Jiangsu Province

Zitieren

APA

Huang, J., Guo, Y., Chen, X., Wei, J., Yi, J., Chen, X., & Chen, L. (2026). HH-MAPPO: A Hierarchical Reinforcement Learning Framework for Dynamic-Scale Target–Attacker–Defender Games. Entropy, 28(7), 793. https://doi.org/10.3390/e28070793

Vancouver

Huang J, Guo Y, Chen X, Wei J, Yi J, Chen X, et al. HH-MAPPO: A Hierarchical Reinforcement Learning Framework for Dynamic-Scale Target–Attacker–Defender Games. Entropy. 2026;28(7):793. doi:10.3390/e28070793

BibTeX
@article{Huang2026,
  author = {Huang, Junhui and Guo, Yan and Chen, Xiliang and Wei, Jianyu and Yi, Jiawei and Chen, Xinliang and Chen, Lifeng},
  title = {HH-MAPPO: A Hierarchical Reinforcement Learning Framework for Dynamic-Scale Target–Attacker–Defender Games},
  journal = {Entropy},
  year = {2026},
  volume = {28},
  number = {7},
  pages = {793},
  doi = {10.3390/e28070793}
}
Kostenlos starten →