EUVIMEDEUVIMED entdecken
Crossref

VulnBench: A Comprehensive Benchmark for Transformer-Based Vulnerability Detection

Jake Norton, David Eyers, Veronica Liesaputra

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (48) · 41649-41651

DOI: 10.1609/aaai.v40i48.42369 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Jake Norton und 2 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Reproducible benchmarking of tools that automatically detect vulnerabilities in source code remains challenging due to inconsistent implementations, varying data preprocessing, and methodological flaws that compromise fair model comparison. In a recent study, 9 in 10 vulnerability detection studies were found to use inappropriate evaluation approaches, with models achieving high scores through spurious correlations rather than actual vulnerability detection. We present VulnBench, an extensible, open-source benchmarking tool that enables fair comparison across models and datasets. Our systematic evaluation of CodeBERT, GraphCodeBERT, CodeT5 (encoder-only and full), and NatGen across eight mostly C/C++ source code datasets reveals that proper threshold optimization can improve F1-scores by up to 54%, as well as wide variation in F1-scores showing the large gap in the difficulty of the vulnerability dataset field. By standardising evaluation protocols, VulnBench enables researchers to distinguish between genuine model improvements and methodological artifacts as well as reducing wasteful duplication of effort spent on reproducing results.

Zitieren

APA

Norton, J., Eyers, D., & Liesaputra, V. (2026). VulnBench: A Comprehensive Benchmark for Transformer-Based Vulnerability Detection. Proceedings of the AAAI Conference on Artificial Intelligence, 40(48), 41649-41651. https://doi.org/10.1609/aaai.v40i48.42369

Vancouver

Norton J, Eyers D, Liesaputra V. VulnBench: A Comprehensive Benchmark for Transformer-Based Vulnerability Detection. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(48):41649-41651. doi:10.1609/aaai.v40i48.42369

BibTeX
@article{Norton2026,
  author = {Norton, Jake and Eyers, David and Liesaputra, Veronica},
  title = {VulnBench: A Comprehensive Benchmark for Transformer-Based Vulnerability Detection},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {48},
  pages = {41649--41651},
  doi = {10.1609/aaai.v40i48.42369}
}
Kostenlos starten →