EUVIMEDEUVIMED entdecken
Crossref

Evaluating API-Level Deep Learning Fuzzers: A Comprehensive Benchmarking Study

Nima Shiri Harzevili, Moshi Wei, Mohammad Mahdi Mohajer, Hung Viet Pham, Song Wang

ACM Transactions on Software Engineering and Methodology · 28.02.2026 · 35 (2) · 1-34

DOI: 10.1145/3729533 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Software Engineering and Methodology, online erschienen am 20.01.2026. Verfasst von Nima Shiri Harzevili und 4 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 88 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In recent years, the practice of fuzzing Deep Learning (DL) APIs has received significant attention in the software engineering community. Many API-level DL fuzzers have been proposed to test individual DL APIs by generating malformed input. Although these fuzzers have been effective in detecting bugs and outperforming prior work, there remains a gap in benchmarking them against ground-truth, real-world bugs in DL libraries. Existing comparisons among these API-level DL fuzzers primarily focus on the bugs detected but do not offer a comprehensive, in-depth evaluation of the fuzzers’ effectiveness. In this work, we perform the first in-depth evaluation of state-of-the-art API-level DL fuzzers that generate tests for single DL APIs, focusing on their effectiveness against real-world bugs. We manually created an extensive benchmark dataset, including 517 real-world DL bugs collected from PyTorch and TensorFlow libraries that can be triggered by malformed inputs. We then apply seven state-of-the-art DL fuzzers— FreeFuzz , DeepRel , NablaFuzz , DocTer , ACETest , TitanFuzz , and FuzzGPT —to our benchmark dataset, following their respective instructions. Our results show that these fuzzers detect only 6.5% (34 out of 517) of the unique real-world bugs in the dataset. Our analysis identifies two dominant factors that impact the effectiveness of these fuzzers in detecting real-world bugs. These findings suggest opportunities for improving the performance of fuzzers in future work. Overall, this study extends previous work on DL fuzzers by providing an extensive evaluation and benchmarking platform for fuzzing DL libraries.

Zitieren

APA

Harzevili, N. S., Wei, M., Mohajer, M. M., Pham, H. V., & Wang, S. (2026). Evaluating API-Level Deep Learning Fuzzers: A Comprehensive Benchmarking Study. ACM Transactions on Software Engineering and Methodology, 35(2), 1-34. https://doi.org/10.1145/3729533

Vancouver

Harzevili NS, Wei M, Mohajer MM, Pham HV, Wang S. Evaluating API-Level Deep Learning Fuzzers: A Comprehensive Benchmarking Study. ACM Transactions on Software Engineering and Methodology. 2026;35(2):1-34. doi:10.1145/3729533

BibTeX
@article{Harzevili2026,
  author = {Harzevili, Nima Shiri and Wei, Moshi and Mohajer, Mohammad Mahdi and Pham, Hung Viet and Wang, Song},
  title = {Evaluating API-Level Deep Learning Fuzzers: A Comprehensive Benchmarking Study},
  journal = {ACM Transactions on Software Engineering and Methodology},
  year = {2026},
  volume = {35},
  number = {2},
  pages = {1--34},
  doi = {10.1145/3729533}
}
Kostenlos starten →