EUVIMEDEUVIMED entdecken
Crossref

HalluCVE: A Multi-Signal Benchmark for Hallucination Detection in LLM-Generated in Cyber Threat Intelligence

Thuan Dao-Duy

Engineering and Technology Horizons · 19.06.2026 · 43 (2)

DOI: 10.55003/eth.430206 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Engineering and Technology Horizons, online erschienen am 19.06.2026. Verfasst von Thuan Dao-Duy.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large Language Models (LLMs) are increasingly utilized for automated Cyber Threat Intelligence (CTI) tasks, such as vulnerability analysis and security advisory generation. However, LLMs are susceptible to hallucination, which refers to the generation of plausible yet factually incorrect content, posing significant risks in security-critical contexts. Although concerns have increased, there is currently no dedicated benchmark for the systematic evaluation of hallucination in LLM-generated cyber threat intelligence (CTI). This study introduces HalluCVE, a multi-signal benchmark designed to detect hallucinations in LLM-generated Common Vulnerabilities and Exposures (CVE). HalluCVE incorporates four complementary detection components: 1) Natural Language Inference-based entailment scoring, 2) lexical factual alignment, 3) LLM-as-a-Judge self-reflection, and 4) cross-model consensus divergence. Five state-of-the-art LLMs are evaluated on 1000 CVE entries as the dataset, from 2022 to 2026, encompassing both known (pre-training cutoff) and unknown (post-cutoff) vulnerabilities. The results indicate pervasive hallucination across all models, with mean Hallucination Index values ranging from 0.480 to 0.820. Notably, models demonstrate near-universal confabulation, reaching up to 100%, when queried about post-cutoff vulnerabilities, and frequently respond with high confidence instead of appropriate refusal. HalluCVE establishes a rigorous evaluation framework for assessing LLM reliability in security-sensitive CTI applications and provides insights into potential mitigation strategies.

Zitieren

APA

Dao-Duy, T. (2026). HalluCVE: A Multi-Signal Benchmark for Hallucination Detection in LLM-Generated in Cyber Threat Intelligence. Engineering and Technology Horizons, 43(2). https://doi.org/10.55003/eth.430206

Vancouver

Dao-Duy T. HalluCVE: A Multi-Signal Benchmark for Hallucination Detection in LLM-Generated in Cyber Threat Intelligence. Engineering and Technology Horizons. 2026;43(2). doi:10.55003/eth.430206

BibTeX
@article{DaoDuy2026,
  author = {Dao-Duy, Thuan},
  title = {HalluCVE: A Multi-Signal Benchmark for Hallucination Detection in LLM-Generated in Cyber Threat Intelligence},
  journal = {Engineering and Technology Horizons},
  year = {2026},
  volume = {43},
  number = {2},
  doi = {10.55003/eth.430206}
}
Kostenlos starten →