EUVIMEDEUVIMED entdecken
Crossref

How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench

Amirali Sajadi, Kostadin Damevski, Preetha Chatterjee

ACM Transactions on Software Engineering and Methodology · 17.11.2026

DOI: 10.1145/3861341 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Software Engineering and Methodology, online erschienen am 17.11.2026. Verfasst von Amirali Sajadi und 2 weiteren Autor:innen. Die Arbeit nennt 99 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) and agentic frameworks are increasingly used for development tasks such as automated program repair (APR). Prior work has identified security risks in LLM-generated code, but most studies focus on synthetic, simplified, or isolated tasks. We present a large-scale security analysis of LLM-generated patches using 20,000+ GitHub issues from SWE-bench. We evaluate developer-written patches, two standalone LLMs (Llama 3.3 Instruct-70B and GPT-5.4 nano), and four agentic frameworks (OpenHands, AutoCodeRover, HoneyComb, and SWE-agent). We also analyze code, issue, and project-level factors associated with insecure patches. Under our evaluated SWE-bench setup, Llama 3.3 introduced 135 manually confirmed vulnerabilities across the train and test sets, compared with 12 in developer-written patches for the same issues. On the SWE-bench test set, GPT-5.4 nano introduced 3 vulnerabilities compared with 2 in developer patches. All four evaluated agentic frameworks also introduced vulnerabilities. OpenHands had the highest retained count, with 44 vulnerabilities across 11 issue instances. Vulnerabilities are associated with broader code modifications and with issues missing specific information. These results suggest that APR systems require proactive risk assessment methods that consider both generated code and issue context.

Zitieren

APA

Sajadi, A., Damevski, K., & Chatterjee, P. (2026). How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench. ACM Transactions on Software Engineering and Methodology. https://doi.org/10.1145/3861341

Vancouver

Sajadi A, Damevski K, Chatterjee P. How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench. ACM Transactions on Software Engineering and Methodology. 2026. doi:10.1145/3861341

BibTeX
@article{Sajadi2026,
  author = {Sajadi, Amirali and Damevski, Kostadin and Chatterjee, Preetha},
  title = {How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench},
  journal = {ACM Transactions on Software Engineering and Methodology},
  year = {2026},
  doi = {10.1145/3861341}
}
Kostenlos starten →