The Best of Both Worlds: Combining Learned Embeddings with Engineered Features for Accurate Prediction of Correct Patches
ACM Transactions on Software Engineering and Methodology · 31.10.2023 · 32 (4) · 1-34
DOI: 10.1145/3576039 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Software Engineering and Methodology, online erschienen am 27.05.2023. Verfasst von Haoye Tian und 9 weiteren Autor:innen. Die Arbeit wurde bisher 36-mal zitiert (Crossref) und nennt 77 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: European Research Council, National Natural Science Foundation of China, Natural Science Foundation of Jiangsu Province, China.
Abstract (Originalsprache)
A large body of the literature on automated program repair develops approaches where patches are automatically generated to be validated against an oracle (e.g., a test suite). Because such an oracle can be imperfect, the generated patches, although validated by the oracle, may actually be incorrect. While the state-of-the-art explores research directions that require dynamic information or rely on manually-crafted heuristics, we study the benefit of learning code representations in order to learn deep features that may encode the properties of patch correctness. Our empirical work investigates different representation learning approaches for code changes to derive embeddings that are amenable to similarity computations of patch correctness identification, and assess the possibility of accurate classification of correct patch by combining learned embeddings with engineered features. Experimental results demonstrate the potential of learned embeddings to empower Leopard (a patch correctness predicting framework implemented in this work) with learning algorithms in reasoning about patch correctness: a machine learning predictor with BERT transformer-based learned embeddings associated with XGBoost achieves an AUC value of about 0.803 in the prediction of patch correctness on a new dataset of 2,147 labeled patches that we collected for the experiments. Our investigations show that deep learned embeddings can lead to complementary/better performance when comparing against the state-of-the-art, PATCH-SIM, which relies on dynamic information. By combining deep learned embeddings and engineered features, Panther (the upgraded version of Leopard implemented in this work) outperforms Leopard with higher scores in terms of AUC, +Recall and -Recall, and can accurately identify more (in)correct patches that cannot be predicted by the classifiers only with learned embeddings or engineered features. Finally, we use an explainable ML technique, SHAP, to empirically interpret how the learned embeddings and engineered features are contributed to the patch correctness prediction.
Förderung
- European Research Council
- National Natural Science Foundation of China
- Natural Science Foundation of Jiangsu Province, China
- Open Project Program of the State Key Laboratory of Mathematical Engineering and Advanced Computing
Zitieren
APA
Tian, H., Liu, K., Li, Y., Kaboré, A. K., Koyuncu, A., Habib, A., Li, L., Wen, J., Klein, J., & Bissyandé, T. F. (2023). The Best of Both Worlds: Combining Learned Embeddings with Engineered Features for Accurate Prediction of Correct Patches. ACM Transactions on Software Engineering and Methodology, 32(4), 1-34. https://doi.org/10.1145/3576039
Vancouver
Tian H, Liu K, Li Y, Kaboré AK, Koyuncu A, Habib A, et al. The Best of Both Worlds: Combining Learned Embeddings with Engineered Features for Accurate Prediction of Correct Patches. ACM Transactions on Software Engineering and Methodology. 2023;32(4):1-34. doi:10.1145/3576039
BibTeX
@article{Tian2023,
author = {Tian, Haoye and Liu, Kui and Li, Yinghua and Kaboré, Abdoul Kader and Koyuncu, Anil and Habib, Andrew and Li, Li and Wen, Junhao and Klein, Jacques and Bissyandé, Tegawendé F.},
title = {The Best of Both Worlds: Combining Learned Embeddings with Engineered Features for Accurate Prediction of Correct Patches},
journal = {ACM Transactions on Software Engineering and Methodology},
year = {2023},
volume = {32},
number = {4},
pages = {1--34},
doi = {10.1145/3576039}
}
EUVIMED