EUVIMEDEUVIMED entdecken
Crossref

Exploring the Capabilities of LLMs for Code-Change-Related Tasks

Lishui Fan, Jiakun Liu, Zhongxin Liu, David Lo, Xin Xia, Shanping Li

ACM Transactions on Software Engineering and Methodology · 31.07.2025 · 34 (6) · 1-36

DOI: 10.1145/3709358 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Software Engineering and Methodology, 2025-07. Verfasst von Lishui Fan und 5 weiteren Autor:innen. Die Arbeit wurde bisher 43-mal zitiert (Crossref) und nennt 111 Literaturangaben. Gefördert von: National Natural Science Foundation of China, National Research Foundation, under its Investigatorship, National Research Foundation, Singapore.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Developers deal with code-change-related tasks daily, e.g., reviewing code. Pre-trained code and code-change-oriented models have been adapted to help developers with such tasks. Recently, large language models (LLMs) have shown their effectiveness in code-related tasks. However, existing LLMs for code focus on general code syntax and semantics rather than the differences between two code versions. Thus, it is an open question how LLMs perform on code-change-related tasks. To answer this question, we conduct an empirical study using \(>\) 1B parameters LLMs on three code-change-related tasks, i.e., code review generation, commit message generation, and just-in-time comment update, with in-context learning (ICL) and parameter-efficient fine-tuning (PEFT, including LoRA and prefix-tuning). We observe that the performance of LLMs is poor without examples and generally improves with examples, but more examples do not always lead to better performance. LLMs tuned with LoRA have comparable performance to the state-of-the-art small pre-trained models. Larger models are not always better, but Llama 2 and Code Llama families are always the best. The best LLMs outperform small pre-trained models on the code changes that only modify comments and perform comparably on other code changes. We suggest future work should focus more on guiding LLMs to learn the knowledge specific to the changes related to code rather than comments for code-change-related tasks.

Förderung

  • National Natural Science Foundation of China
  • National Research Foundation, under its Investigatorship
  • National Research Foundation, Singapore
  • Zhejiang University Education Foundation Qizhen Scholar Foundation

Zitieren

APA

Fan, L., Liu, J., Liu, Z., Lo, D., Xia, X., & Li, S. (2025). Exploring the Capabilities of LLMs for Code-Change-Related Tasks. ACM Transactions on Software Engineering and Methodology, 34(6), 1-36. https://doi.org/10.1145/3709358

Vancouver

Fan L, Liu J, Liu Z, Lo D, Xia X, Li S. Exploring the Capabilities of LLMs for Code-Change-Related Tasks. ACM Transactions on Software Engineering and Methodology. 2025;34(6):1-36. doi:10.1145/3709358

BibTeX
@article{Fan2025,
  author = {Fan, Lishui and Liu, Jiakun and Liu, Zhongxin and Lo, David and Xia, Xin and Li, Shanping},
  title = {Exploring the Capabilities of LLMs for Code-Change-Related Tasks},
  journal = {ACM Transactions on Software Engineering and Methodology},
  year = {2025},
  volume = {34},
  number = {6},
  pages = {1--36},
  doi = {10.1145/3709358}
}
Kostenlos starten →