HalluCVE: A Multi-Signal Benchmark for Hallucination Detection in LLM-Generated in Cyber Threat Intelligence
Engineering and Technology Horizons · 19.06.2026 · 43 (2)
DOI: 10.55003/eth.430206 ↗Kurz gefasst
Fachartikel in der Zeitschrift Engineering and Technology Horizons, online erschienen am 19.06.2026. Verfasst von Thuan Dao-Duy.
Abstract (Originalsprache)
Large Language Models (LLMs) are increasingly utilized for automated Cyber Threat Intelligence (CTI) tasks, such as vulnerability analysis and security advisory generation. However, LLMs are susceptible to hallucination, which refers to the generation of plausible yet factually incorrect content, posing significant risks in security-critical contexts. Although concerns have increased, there is currently no dedicated benchmark for the systematic evaluation of hallucination in LLM-generated cyber threat intelligence (CTI). This study introduces HalluCVE, a multi-signal benchmark designed to detect hallucinations in LLM-generated Common Vulnerabilities and Exposures (CVE). HalluCVE incorporates four complementary detection components: 1) Natural Language Inference-based entailment scoring, 2) lexical factual alignment, 3) LLM-as-a-Judge self-reflection, and 4) cross-model consensus divergence. Five state-of-the-art LLMs are evaluated on 1000 CVE entries as the dataset, from 2022 to 2026, encompassing both known (pre-training cutoff) and unknown (post-cutoff) vulnerabilities. The results indicate pervasive hallucination across all models, with mean Hallucination Index values ranging from 0.480 to 0.820. Notably, models demonstrate near-universal confabulation, reaching up to 100%, when queried about post-cutoff vulnerabilities, and frequently respond with high confidence instead of appropriate refusal. HalluCVE establishes a rigorous evaluation framework for assessing LLM reliability in security-sensitive CTI applications and provides insights into potential mitigation strategies.
Zitieren
APA
Dao-Duy, T. (2026). HalluCVE: A Multi-Signal Benchmark for Hallucination Detection in LLM-Generated in Cyber Threat Intelligence. Engineering and Technology Horizons, 43(2). https://doi.org/10.55003/eth.430206
Vancouver
Dao-Duy T. HalluCVE: A Multi-Signal Benchmark for Hallucination Detection in LLM-Generated in Cyber Threat Intelligence. Engineering and Technology Horizons. 2026;43(2). doi:10.55003/eth.430206
BibTeX
@article{DaoDuy2026,
author = {Dao-Duy, Thuan},
title = {HalluCVE: A Multi-Signal Benchmark for Hallucination Detection in LLM-Generated in Cyber Threat Intelligence},
journal = {Engineering and Technology Horizons},
year = {2026},
volume = {43},
number = {2},
doi = {10.55003/eth.430206}
}
EUVIMED