EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Today’s Cat Is Tomorrow’s Dog: Accounting for Time-Based Changes in the Labels of ML Vulnerability Detection Approaches

Ranindya Paramitha, Yuan Feng, Fabio Massacci

Proceedings of the ACM on Software Engineering · 19.06.2025 · 2 (FSE) · 335-357

DOI: 10.1145/3715731 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the ACM on Software Engineering, online erschienen am 19.06.2025. Verfasst von Ranindya Paramitha und 2 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 69 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: H2020 LEIT Information and Communication Technologies, HORIZON EUROPE Civil security for society, Dutch Research Council.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Vulnerability datasets used for ML testing implicitly contain retrospective information. When tested on the field, one can only use the labels available at the time of training and testing (e.g. seen and assumed negatives). As vulnerabilities are discovered across calendar time, labels change and past performance is not necessarily aligned with future performance. Past works only considered the slices of the whole history (e.g. DiverseVUl) or individual differences between releases (e.g. Jimenez et al. ESEC/FSE 2019). Such approaches are either too optimistic in training (e.g. the whole history) or too conservative (e.g. consecutive releases). We propose a method to restructure a dataset into a series of datasets in which both training and testing labels change to account for the knowledge available at the time. If the model is actually learning, it should improve its performance over time as more data becomes available and data becomes more stable, an effect that can be checked with the Mann-Kendall test. We validate our methodology for vulnerability detection with 4 time-based datasets (3 projects from BigVul dataset + Vuldeepecker’s NVD) and 5 ML models (Code2Vec, CodeBERT, LineVul, ReGVD, and Vuldeepecker). In contrast to the intuitive expectation (more retrospective information, better performance), the trend results show that performance changes inconsistently across the years, showing that most models are not learning.

Förderung

  • H2020 LEIT Information and Communication Technologies
  • HORIZON EUROPE Civil security for society
  • Dutch Research Council

Zitieren

APA

Paramitha, R., Feng, Y., & Massacci, F. (2025). Today’s Cat Is Tomorrow’s Dog: Accounting for Time-Based Changes in the Labels of ML Vulnerability Detection Approaches. Proceedings of the ACM on Software Engineering, 2(FSE), 335-357. https://doi.org/10.1145/3715731

Vancouver

Paramitha R, Feng Y, Massacci F. Today’s Cat Is Tomorrow’s Dog: Accounting for Time-Based Changes in the Labels of ML Vulnerability Detection Approaches. Proceedings of the ACM on Software Engineering. 2025;2(FSE):335-357. doi:10.1145/3715731

BibTeX
@article{Paramitha2025,
  author = {Paramitha, Ranindya and Feng, Yuan and Massacci, Fabio},
  title = {Today’s Cat Is Tomorrow’s Dog: Accounting for Time-Based Changes in the Labels of ML Vulnerability Detection Approaches},
  journal = {Proceedings of the ACM on Software Engineering},
  year = {2025},
  volume = {2},
  number = {FSE},
  pages = {335--357},
  doi = {10.1145/3715731}
}
Kostenlos starten →