EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

SD2AC: A reinforcement learning framework using distribution evaluation and sequential decision-making for UCAV combat

Tao Yang, Xinhao Shi, Cheng Xu, Yulin Yang, Hongzhe Liu, Qinghan Zeng

Journal of Computational Design and Engineering · 11.07.2025 · 12 (7) · 96-112

DOI: 10.1093/jcde/qwaf045 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Computational Design and Engineering, online erschienen am 25.04.2025. Verfasst von Tao Yang und 5 weiteren Autor:innen. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 40 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: National Language Commission, National Natural Science Foundation of China, Beijing Municipal Education Commission.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The increasing complexity of autonomous drone swarm operations, particularly in adversarial air combat scenarios, presents significant challenges in multi-agent reinforcement learning (MARL). Existing approaches often face issues such as poor coordination, suboptimal convergence, and overestimation bias, undermining their applicability to high-stakes environments. We propose the Sequential Decision Distribution Soft Actor-Critic (SD2AC), a novel off-policy MARL framework for drone swarm coordination to address these challenges. SD2AC introduces three key innovations: (1) a sequential decision-making framework in which agents adaptively condition their decisions on the policies of preceding agents, enhancing coordination and reducing conflicts; (2) a distributional Q-value critic that models the full return distribution to mitigate overestimation bias and improve policy robustness; and (3) an adaptive twin value distribution learning mechanism that leverages dual critics to dynamically select conservative value estimates, ensuring stable learning under uncertainty. Rigorous evaluation in diverse environments, including Close Air Combat, Multi-agent Encirclement with Collision Avoidance (MECA), and MAMuJoCo, demonstrate the superiority of SD2AC over state-of-the-art MARL methods in terms of reward optimization, convergence speed, and coordination efficiency. Ablation studies further validate the individual contributions of each component.

Förderung

  • National Language Commission
  • National Natural Science Foundation of China
  • Beijing Municipal Education Commission
  • High-Level Innovative Teams in Beijing Municipal Institutions
  • Beijing Natural Science Foundation
  • Beijing Union University

Zitieren

APA

Yang, T., Shi, X., Xu, C., Yang, Y., Liu, H., & Zeng, Q. (2025). SD2AC: A reinforcement learning framework using distribution evaluation and sequential decision-making for UCAV combat. Journal of Computational Design and Engineering, 12(7), 96-112. https://doi.org/10.1093/jcde/qwaf045

Vancouver

Yang T, Shi X, Xu C, Yang Y, Liu H, Zeng Q. SD2AC: A reinforcement learning framework using distribution evaluation and sequential decision-making for UCAV combat. Journal of Computational Design and Engineering. 2025;12(7):96-112. doi:10.1093/jcde/qwaf045

BibTeX
@article{Yang2025,
  author = {Yang, Tao and Shi, Xinhao and Xu, Cheng and Yang, Yulin and Liu, Hongzhe and Zeng, Qinghan},
  title = {SD2AC: A reinforcement learning framework using distribution evaluation and sequential decision-making for UCAV combat},
  journal = {Journal of Computational Design and Engineering},
  year = {2025},
  volume = {12},
  number = {7},
  pages = {96--112},
  doi = {10.1093/jcde/qwaf045}
}
Kostenlos starten →