EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Evaluating Large Language Models in Cybersecurity: A Systematic Taxonomy and Empirical Analysis

Mantun Chen, Hua Cheng, Ting Su, Minghui Chen, Wenjun Cai, Hongcheng Zou

Electronics · 21.05.2026 · 15 (10) · 2222

DOI: 10.3390/electronics15102222 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Electronics, online erschienen am 21.05.2026. Verfasst von Mantun Chen und 5 weiteren Autor:innen. Die Arbeit nennt 124 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This paper presents a Systematization of Knowledge (SoK) on the evaluation methodologies and capability boundaries of Large Language Models (LLMs) in cybersecurity. We propose a Three-Dimensional Taxonomy Matrix to systematize existing metrics across offensive domains, defensive applications, and inherent architectural flaws. Beyond categorization, this matrix functions as a predictive framework to expose structural evaluation blind spots. Specifically, by intersecting target domains with failure attributions, it identifies a critical, unresolved frontier: measuring cross-architecture semantic equivalence in low-level reverse engineering. Empirically, synthesizing 39 frontier benchmarks reveals a systemic evaluation gap: static metric success rarely translates into end-to-end adversarial efficacy. In offensive domains, high penetration rates correlate strongly with pre-training data contamination. When subjected to semantics-preserving code obfuscation as a stress test, zero-shot, tool-free exploit success rates collapse to near 0%. In defensive contexts, cross-procedural code auditing struggles, yielding a peak F1-score of only 23.83%. Furthermore, models suffer from over-alignment-induced functional degradation, with joint-testing frameworks recording up to a 77% functional loss in automated program repair. Our analysis strongly suggests that purely autoregressive mechanisms drive severe technical hallucinations, evidenced by a 19.7% package dependency fabrication rate. Evaluations also expose significant attack surfaces and a significant safety-utility tradeoff: models succumb to prompt leakage attacks at rates up to 86.2%, while heavily aligned versions simultaneously exhibit excessively high False Refusal Rates (FRR) for benign, borderline security queries. Finally, we delineate a theoretical neuro-symbolic roadmap—integrating LLM heuristics with deterministic formal methods—to structurally mitigate the limitations of the autoregressive paradigm.

Zitieren

APA

Chen, M., Cheng, H., Su, T., Chen, M., Cai, W., & Zou, H. (2026). Evaluating Large Language Models in Cybersecurity: A Systematic Taxonomy and Empirical Analysis. Electronics, 15(10), 2222. https://doi.org/10.3390/electronics15102222

Vancouver

Chen M, Cheng H, Su T, Chen M, Cai W, Zou H. Evaluating Large Language Models in Cybersecurity: A Systematic Taxonomy and Empirical Analysis. Electronics. 2026;15(10):2222. doi:10.3390/electronics15102222

BibTeX
@article{Chen2026,
  author = {Chen, Mantun and Cheng, Hua and Su, Ting and Chen, Minghui and Cai, Wenjun and Zou, Hongcheng},
  title = {Evaluating Large Language Models in Cybersecurity: A Systematic Taxonomy and Empirical Analysis},
  journal = {Electronics},
  year = {2026},
  volume = {15},
  number = {10},
  pages = {2222},
  doi = {10.3390/electronics15102222}
}
Kostenlos starten →