VulnBench: A Comprehensive Benchmark for Transformer-Based Vulnerability Detection
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (48) · 41649-41651
DOI: 10.1609/aaai.v40i48.42369 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Jake Norton und 2 weiteren Autor:innen.
Abstract (Originalsprache)
Reproducible benchmarking of tools that automatically detect vulnerabilities in source code remains challenging due to inconsistent implementations, varying data preprocessing, and methodological flaws that compromise fair model comparison. In a recent study, 9 in 10 vulnerability detection studies were found to use inappropriate evaluation approaches, with models achieving high scores through spurious correlations rather than actual vulnerability detection. We present VulnBench, an extensible, open-source benchmarking tool that enables fair comparison across models and datasets. Our systematic evaluation of CodeBERT, GraphCodeBERT, CodeT5 (encoder-only and full), and NatGen across eight mostly C/C++ source code datasets reveals that proper threshold optimization can improve F1-scores by up to 54%, as well as wide variation in F1-scores showing the large gap in the difficulty of the vulnerability dataset field. By standardising evaluation protocols, VulnBench enables researchers to distinguish between genuine model improvements and methodological artifacts as well as reducing wasteful duplication of effort spent on reproducing results.
Zitieren
APA
Norton, J., Eyers, D., & Liesaputra, V. (2026). VulnBench: A Comprehensive Benchmark for Transformer-Based Vulnerability Detection. Proceedings of the AAAI Conference on Artificial Intelligence, 40(48), 41649-41651. https://doi.org/10.1609/aaai.v40i48.42369
Vancouver
Norton J, Eyers D, Liesaputra V. VulnBench: A Comprehensive Benchmark for Transformer-Based Vulnerability Detection. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(48):41649-41651. doi:10.1609/aaai.v40i48.42369
BibTeX
@article{Norton2026,
author = {Norton, Jake and Eyers, David and Liesaputra, Veronica},
title = {VulnBench: A Comprehensive Benchmark for Transformer-Based Vulnerability Detection},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {48},
pages = {41649--41651},
doi = {10.1609/aaai.v40i48.42369}
}
EUVIMED