Diagnosing Planning Quality in LLM-Based Penetration Testing Using Execution-Free Evaluation
Journal of Smart Algorithms and Applications (JSAA) · 24.04.2026 · 3 (1) · 51-70
DOI: 10.66279/enzxq198 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Smart Algorithms and Applications (JSAA), online erschienen am 24.04.2026. Verfasst von Howaida Allam und 1 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Evaluating language model (LLM) penetration testing agents through execution conflates planning quality with environmental factors, including network latency, tool failures, and infrastructure variability. Analysis of AutoPenBench logs reveals that 40\% of agent failures arise from environmental timeouts rather than planning errors, obscuring the true diagnostic signal. We present an \textit{execution-free}, rubric-based evaluation framework that directly assesses the quality of LLM-generated attack plans along five independently scored dimensions: tool relevance, sequence logic, completeness, efficiency, and reasoning quality. Ten penetration testing scenarios spanning reconnaissance to post-exploitation are stratified by difficulty and evaluated across four models: GPT-4o, GPT-4o-mini, Llama-3.3-70B, and Qwen-2.5-14B—using automated scoring via Claude Sonnet 4.5 at temperature=0. Model scores range from 58\% to 74\%, substantially above the random baseline of 18\% but below the 96\% achieved by expert-generated reference plans, confirming a 78-percentage-point discriminative range. Dimension-level analysis reveals balanced performance across rubric criteria (means 7.0–7.3/10) with notable inter-model variation in efficiency (6.0–7.6) and reasoning quality (5.8–8.1). To strengthen validity, we discuss requirements for human expert validation and inter-rater reliability (target Krippendorff's $\alpha \geq 0.67$), multi-sample generation for variance estimation, and the inclusion of evasion and obfuscation scenarios. The framework offers a reproducible, infrastructure-independent diagnostic tool for model development and security education, intended to complement—not replace—execution-based benchmarks.
Zitieren
APA
Allam, H., & Guezzaz, A. (2026). Diagnosing Planning Quality in LLM-Based Penetration Testing Using Execution-Free Evaluation. Journal of Smart Algorithms and Applications (JSAA), 3(1), 51-70. https://doi.org/10.66279/enzxq198
Vancouver
Allam H, Guezzaz A. Diagnosing Planning Quality in LLM-Based Penetration Testing Using Execution-Free Evaluation. Journal of Smart Algorithms and Applications (JSAA). 2026;3(1):51-70. doi:10.66279/enzxq198
BibTeX
@article{Allam2026,
author = {Allam, Howaida and Guezzaz, Azidine},
title = {Diagnosing Planning Quality in LLM-Based Penetration Testing Using Execution-Free Evaluation},
journal = {Journal of Smart Algorithms and Applications (JSAA)},
year = {2026},
volume = {3},
number = {1},
pages = {51--70},
doi = {10.66279/enzxq198}
}
EUVIMED