EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Understanding Automated Program Repair Agents through the Lens of Traceability: An Empirical Study

Ira Ceka, Hailie Mitchell, Saurabh Pujar, Luca Buratti, Shyam Ramji, Junfeng Yang, Gail Kaiser, Baishakhi Ray

Proceedings of the ACM on Software Engineering · 01.10.2026 · 3 (ISSTA) · 4450-4473

DOI: 10.1145/3832285 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the ACM on Software Engineering, online erschienen am 01.10.2026. Verfasst von Ira Ceka und 7 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 62 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NSF.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Automated Program Repair (APR) agents leverage large language models (LLMs) to autonomously diagnose and patch software bugs using planning, reasoning, and tools. Although these agents show strong performance on leaderboards such as SWE-bench, little is understood about how they take actions, where they fail, and how their behavior compares to human developers. In this paper, we present the first systematic analysis of these limitations using 5 state-of-the-art APR agents. We trace the full decision-making pipelines of the 5 APR agents across 500 real-world repair tasks, from issue description to patch validation. Our study reveals that, while agents excel at simple fixes, they struggle with logic-intensive bugs, often generating verbose, overfitted patches that pass existing test suites without solving the root cause. Test generation and regression test selection remain major bottlenecks, as agents fail to reproduce issues or run relevant regression tests. Moreover, many agents operate with primitive tooling (e.g. bash scripts) and do not have access to debuggers or program analysis tools. These findings highlight key limitations of current APR systems and motivate several directions for next-generation APR design, including but not limited to: (1) a shift-left approach emphasizing early, high-quality test generation and validation to reduce spurious fixes and improve semantic correctness; (2) richer, more integrated tool ecosystems; (3) diversified agent architectures that combine complementary strengths; and (4) benchmarks that prioritize semantic repair quality and test-generation fidelity over surface-level success metrics.

Förderung

  • NSF

Zitieren

APA

Ceka, I., Mitchell, H., Pujar, S., Buratti, L., Ramji, S., Yang, J., Kaiser, G., & Ray, B. (2026). Understanding Automated Program Repair Agents through the Lens of Traceability: An Empirical Study. Proceedings of the ACM on Software Engineering, 3(ISSTA), 4450-4473. https://doi.org/10.1145/3832285

Vancouver

Ceka I, Mitchell H, Pujar S, Buratti L, Ramji S, Yang J, et al. Understanding Automated Program Repair Agents through the Lens of Traceability: An Empirical Study. Proceedings of the ACM on Software Engineering. 2026;3(ISSTA):4450-4473. doi:10.1145/3832285

BibTeX
@article{Ceka2026,
  author = {Ceka, Ira and Mitchell, Hailie and Pujar, Saurabh and Buratti, Luca and Ramji, Shyam and Yang, Junfeng and Kaiser, Gail and Ray, Baishakhi},
  title = {Understanding Automated Program Repair Agents through the Lens of Traceability: An Empirical Study},
  journal = {Proceedings of the ACM on Software Engineering},
  year = {2026},
  volume = {3},
  number = {ISSTA},
  pages = {4450--4473},
  doi = {10.1145/3832285}
}
Kostenlos starten →