EUVIMEDEUVIMED entdecken
Crossref

Automatically Recommend Code Updates: Are We There Yet?

Yue Liu, Chakkrit Tantithamthavorn, Yonghui Liu, Patanamon Thongtanunam, Li Li

ACM Transactions on Software Engineering and Methodology · 30.11.2024 · 33 (8) · 1-27

DOI: 10.1145/3678167 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Software Engineering and Methodology, online erschienen am 03.12.2024. Verfasst von Yue Liu und 4 weiteren Autor:innen. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 102 Literaturangaben. Gefördert von: Australian Research Council’s Discovery Early Career Researcher.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In recent years, large pre-trained Language Models of Code (CodeLMs) have shown promising results on various software engineering tasks. One such task is automatic code update recommendation, which transforms outdated code snippets into their approved and revised counterparts. Although many CodeLM-based approaches have been proposed, claiming high accuracy, their effectiveness and reliability on real-world code update tasks remain questionable. In this article, we present the first extensive evaluation of state-of-the-art CodeLMs for automatically recommending code updates. We assess their performance on two diverse datasets of paired updated methods, considering factors such as temporal evolution, project specificity, method size, and update complexity. Our results reveal that while CodeLMs exhibit higher performance in settings that ignore temporal information, they struggle in more realistic time-wise scenarios and generalize poorly to new projects. Furthermore, CodeLM performance decreases significantly for larger methods and more complex updates. Furthermore, we observe that many CodeLM-generated “updates” are actually null, especially in time-wise settings, and meaningful edits remain challenging. Our findings highlight the significant gap between the perceived and actual effectiveness of CodeLMs for real-world code update recommendation and emphasize the need for more research on improving their practicality, robustness, and generalizability.

Förderung

  • Australian Research Council’s Discovery Early Career Researcher

Zitieren

APA

Liu, Y., Tantithamthavorn, C., Liu, Y., Thongtanunam, P., & Li, L. (2024). Automatically Recommend Code Updates: Are We There Yet?. ACM Transactions on Software Engineering and Methodology, 33(8), 1-27. https://doi.org/10.1145/3678167

Vancouver

Liu Y, Tantithamthavorn C, Liu Y, Thongtanunam P, Li L. Automatically Recommend Code Updates: Are We There Yet?. ACM Transactions on Software Engineering and Methodology. 2024;33(8):1-27. doi:10.1145/3678167

BibTeX
@article{Liu2024,
  author = {Liu, Yue and Tantithamthavorn, Chakkrit and Liu, Yonghui and Thongtanunam, Patanamon and Li, Li},
  title = {Automatically Recommend Code Updates: Are We There Yet?},
  journal = {ACM Transactions on Software Engineering and Methodology},
  year = {2024},
  volume = {33},
  number = {8},
  pages = {1--27},
  doi = {10.1145/3678167}
}
Kostenlos starten →