Unprecedented Code Change Automation: The Fusion of LLMs and Transformation by Example
Proceedings of the ACM on Software Engineering · 12.07.2024 · 1 (FSE) · 631-653
DOI: 10.1145/3643755 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the ACM on Software Engineering, online erschienen am 12.07.2024. Verfasst von Malinda Dilhara und 3 weiteren Autor:innen. Die Arbeit wurde bisher 26-mal zitiert (Crossref) und nennt 67 Literaturangaben. Gefördert von: NSF.
Abstract (Originalsprache)
Software developers often repeat the same code changes within a project or across different projects. These repetitive changes are known as “code change patterns” (CPATs). Automating CPATs is crucial to expedite the software development process. While current Transformation by Example (TBE) techniques can automate CPATs, they are limited by the quality and quantity of the provided input examples. Thus, they miss transforming code variations that do not have the exact syntax, data-, or control-flow of the provided input examples, despite being semantically similar. Large Language Models (LLMs), pre-trained on extensive source code datasets, offer a potential solution. Harnessing the capability of LLMs to generate semantically equivalent, yet previously unseen variants of the original CPAT could significantly increase the effectiveness of TBE systems. In this paper, we first discover best practices for harnessing LLMs to generate code variants that meet three criteria: correctness (semantic equivalence to the original CPAT), usefulness (reflecting what developers typically write), and applicability (aligning with the primary intent of the original CPAT). We then implement these practices in our tool PyCraft, which synergistically combines static code analysis, dynamic analysis, and LLM capabilities. By employing chain-of-thought reasoning, PyCraft generates variations of input examples and comprehensive test cases that identify correct variations with an F-measure of 96.6%. Our algorithm uses feedback iteration to expand the original input examples by an average factor of 58x. Using these richly generated examples, we inferred transformation rules and then automated these changes, resulting in an increase of up to 39x, with an average increase of 14x in target codes compared to a previous state-of-the-art tool that relies solely on static analysis. We submitted patches generated by PyCraft to a range of projects, notably esteemed ones like microsoft/DeepSpeed and IBM/inFairness. Their developers accepted and merged 83% the 86 CPAT instances submitted through 44 pull requests. This confirms the usefulness of these changes.
Förderung
- NSF
Zitieren
APA
Dilhara, M., Bellur, A., Bryksin, T., & Dig, D. (2024). Unprecedented Code Change Automation: The Fusion of LLMs and Transformation by Example. Proceedings of the ACM on Software Engineering, 1(FSE), 631-653. https://doi.org/10.1145/3643755
Vancouver
Dilhara M, Bellur A, Bryksin T, Dig D. Unprecedented Code Change Automation: The Fusion of LLMs and Transformation by Example. Proceedings of the ACM on Software Engineering. 2024;1(FSE):631-653. doi:10.1145/3643755
BibTeX
@article{Dilhara2024,
author = {Dilhara, Malinda and Bellur, Abhiram and Bryksin, Timofey and Dig, Danny},
title = {Unprecedented Code Change Automation: The Fusion of LLMs and Transformation by Example},
journal = {Proceedings of the ACM on Software Engineering},
year = {2024},
volume = {1},
number = {FSE},
pages = {631--653},
doi = {10.1145/3643755}
}
EUVIMED