Evaluating the Effectiveness of Deep Learning Models for Foundational Program Analysis Tasks
ACM Transactions on Software Engineering and Methodology · 31.12.2026 · 35 (12) · 1-42
DOI: 10.1145/3802541 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Software Engineering and Methodology, online erschienen am 18.11.2026. Verfasst von Qian Chen und 7 weiteren Autor:innen. Die Arbeit nennt 96 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, National Key R&D, Basic Research Program of Jiangsu.
Abstract (Originalsprache)
While deep neural networks provide state-of-the-art solutions to a wide range of programming language tasks, their effectiveness in dealing with foundational program analysis tasks remains under explored. In this article, we present an empirical study that evaluates four prominent models of code (i.e., CuBERT, CodeBERT, GGNN, and Graph Sandwiches), plus four popular large language models (i.e., GPT3.5 , GPT-4o mini , Qwen2.5-Coder , and DeepSeek Coder ), in two such foundational tasks: (1) alias prediction, in which models predict whether two pointers must alias, may alias, or must not alias; and (2) equivalence prediction, in which models predict whether or not two programs are semantically equivalent. At the core of this study is CodeSem , a dataset built upon the source code of real-world flagship software (e.g., Linux Kernel, GCC, MySQL) and manually validated for the two prediction tasks. Results show that all models are accurate in both prediction tasks. We also conduct a comprehensive, in-depth analysis of the results of all models in both tasks, concluding that deep learning models are generally capable of performing foundational tasks in program analysis, even though in specific cases their weaknesses are also evident. Our code and evaluation data are publicly available at https://github.com/CodeSemDataset/CodeSem .
Förderung
- National Natural Science Foundation of China
- National Key R&D
- Basic Research Program of Jiangsu
- China Postdoctoral Science Foundation
- Collaborative Innovation Center of Novel Software Technology and Industrialization, Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Mini…
- Program for Outstanding PhD Candidates of Nanjing University
Zitieren
APA
Chen, Q., Chen, R., Yu, C., Liu, R., Zhang, C., Wang, Y., Wang, K., & Wang, L. (2026). Evaluating the Effectiveness of Deep Learning Models for Foundational Program Analysis Tasks. ACM Transactions on Software Engineering and Methodology, 35(12), 1-42. https://doi.org/10.1145/3802541
Vancouver
Chen Q, Chen R, Yu C, Liu R, Zhang C, Wang Y, et al. Evaluating the Effectiveness of Deep Learning Models for Foundational Program Analysis Tasks. ACM Transactions on Software Engineering and Methodology. 2026;35(12):1-42. doi:10.1145/3802541
BibTeX
@article{Chen2026,
author = {Chen, Qian and Chen, Rouyi and Yu, Chenyang and Liu, Ruyan and Zhang, Chi and Wang, Yu and Wang, Ke and Wang, Linzhang},
title = {Evaluating the Effectiveness of Deep Learning Models for Foundational Program Analysis Tasks},
journal = {ACM Transactions on Software Engineering and Methodology},
year = {2026},
volume = {35},
number = {12},
pages = {1--42},
doi = {10.1145/3802541}
}
EUVIMED