A hierarchical reinforcement learning system for efficient and scalable network penetration testing
The Computer Journal · 18.08.2026
DOI: 10.1093/comjnl/bxag086 ↗Kurz gefasst
Fachartikel in der Zeitschrift The Computer Journal, online erschienen am 18.08.2026. Verfasst von Dongze Wei und 7 weiteren Autor:innen. Die Arbeit nennt 49 Literaturangaben. Gefördert von: Natural Science Foundation of Henan Province of China.
Abstract (Originalsprache)
Penetration testing (PT) is a critical method for assessing network security by simulating cyberattacks to identify vulnerabilities. While reinforcement learning has been applied to automate PT, modeling it as a Markov decision process often leads to state-space complexity and dimensional explosion. To address these problems, we propose the Hierarchical Advantage Memory-enhanced Deep Q-Network (HAML-DQN) algorithm, an intelligent PT framework that integrates hierarchical reinforcement learning (HRL) with memory enhancement and action masking (AM) mechanisms. The HRL architecture decomposes PT tasks into high-level host selection and low-level attack execution, effectively mitigating the curse of dimensionality in state and action spaces. The memory-enhanced module stores high-reward action sequences to guide exploration, while the AM mechanism filters out redundant or illogical actions, thus imitating expert decision making. Our contributions include: (i) designing an HRL framework for PT, (ii) improved memory-enhanced exploration strategy, and (iii) proposed an action masking (AM) mechanism for efficient path planning. The results highlight the potential of HAML-DQN to advance automated PT and enhance cybersecurity defenses. Experiments conducted on the Network Attack Simulator platform demonstrate that the HAML-DQN method significantly enhances PT efficiency and exhibits strong scalability in large-scale networks, increasing the applicable network scale by nearly 20 times compared with the baseline DQN algorithm.
Förderung
- Natural Science Foundation of Henan Province of China
Zitieren
APA
Wei, D., Zhou, T., Liu, J., Zhu, J., Li, Z., Song, M., Li, D., & Zhu, X. (2026). A hierarchical reinforcement learning system for efficient and scalable network penetration testing. The Computer Journal. https://doi.org/10.1093/comjnl/bxag086
Vancouver
Wei D, Zhou T, Liu J, Zhu J, Li Z, Song M, et al. A hierarchical reinforcement learning system for efficient and scalable network penetration testing. The Computer Journal. 2026. doi:10.1093/comjnl/bxag086
BibTeX
@article{Wei2026,
author = {Wei, Dongze and Zhou, Tianyang and Liu, Jinghu and Zhu, Junhu and Li, Zhi and Song, Mengbo and Li, Dongyang and Zhu, Xiaodong},
title = {A hierarchical reinforcement learning system for efficient and scalable network penetration testing},
journal = {The Computer Journal},
year = {2026},
doi = {10.1093/comjnl/bxag086}
}
EUVIMED