Today’s Cat Is Tomorrow’s Dog: Accounting for Time-Based Changes in the Labels of ML Vulnerability Detection Approaches
Proceedings of the ACM on Software Engineering · 19.06.2025 · 2 (FSE) · 335-357
DOI: 10.1145/3715731 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the ACM on Software Engineering, online erschienen am 19.06.2025. Verfasst von Ranindya Paramitha und 2 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 69 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: H2020 LEIT Information and Communication Technologies, HORIZON EUROPE Civil security for society, Dutch Research Council.
Abstract (Originalsprache)
Vulnerability datasets used for ML testing implicitly contain retrospective information. When tested on the field, one can only use the labels available at the time of training and testing (e.g. seen and assumed negatives). As vulnerabilities are discovered across calendar time, labels change and past performance is not necessarily aligned with future performance. Past works only considered the slices of the whole history (e.g. DiverseVUl) or individual differences between releases (e.g. Jimenez et al. ESEC/FSE 2019). Such approaches are either too optimistic in training (e.g. the whole history) or too conservative (e.g. consecutive releases). We propose a method to restructure a dataset into a series of datasets in which both training and testing labels change to account for the knowledge available at the time. If the model is actually learning, it should improve its performance over time as more data becomes available and data becomes more stable, an effect that can be checked with the Mann-Kendall test. We validate our methodology for vulnerability detection with 4 time-based datasets (3 projects from BigVul dataset + Vuldeepecker’s NVD) and 5 ML models (Code2Vec, CodeBERT, LineVul, ReGVD, and Vuldeepecker). In contrast to the intuitive expectation (more retrospective information, better performance), the trend results show that performance changes inconsistently across the years, showing that most models are not learning.
Förderung
- H2020 LEIT Information and Communication Technologies
- HORIZON EUROPE Civil security for society
- Dutch Research Council
Zitieren
APA
Paramitha, R., Feng, Y., & Massacci, F. (2025). Today’s Cat Is Tomorrow’s Dog: Accounting for Time-Based Changes in the Labels of ML Vulnerability Detection Approaches. Proceedings of the ACM on Software Engineering, 2(FSE), 335-357. https://doi.org/10.1145/3715731
Vancouver
Paramitha R, Feng Y, Massacci F. Today’s Cat Is Tomorrow’s Dog: Accounting for Time-Based Changes in the Labels of ML Vulnerability Detection Approaches. Proceedings of the ACM on Software Engineering. 2025;2(FSE):335-357. doi:10.1145/3715731
BibTeX
@article{Paramitha2025,
author = {Paramitha, Ranindya and Feng, Yuan and Massacci, Fabio},
title = {Today’s Cat Is Tomorrow’s Dog: Accounting for Time-Based Changes in the Labels of ML Vulnerability Detection Approaches},
journal = {Proceedings of the ACM on Software Engineering},
year = {2025},
volume = {2},
number = {FSE},
pages = {335--357},
doi = {10.1145/3715731}
}
EUVIMED