EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Fully Autonomous Programming Using Iterative Multi-Agent Debugging with Large Language Models

Anastasiia Grishina, Vadim Liventsev, Aki Härmä, Leon Moonen

ACM Transactions on Evolutionary Learning and Optimization · 31.03.2025 · 5 (1) · 1-37

DOI: 10.1145/3719351 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Evolutionary Learning and Optimization, online erschienen am 29.03.2025. Verfasst von Anastasiia Grishina und 3 weiteren Autor:innen. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 88 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: European Commission through Horizon 2020, Research Council of Norway through the secureIT, Experimental Infrastructure for Exploration of Exascale Computing.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Program synthesis with Large Language Models (LLMs) suffers from a “near-miss syndrome”: The generated code closely resembles a correct solution but fails unit tests due to minor errors. We address this with a multi-agent framework called Synthesize, Execute, Instruct, Debug, and Repair (SEIDR). Effectively applying SEIDR to instruction-tuned LLMs requires determining (a) optimal prompts for LLMs, (b) what ranking algorithm selects the best programs in debugging rounds, and (c) balancing the repair of unsuccessful programs with the generation of new ones. We empirically explore these tradeoffs by comparing replace-focused, repair-focused, and hybrid debug strategies. We also evaluate lexicase and tournament selection to rank candidates in each generation. On Program Synthesis Benchmark 2 (PSB2), our framework outperforms both conventional use of OpenAI Codex without a repair phase and traditional genetic programming approaches. SEIDR outperforms the use of an LLM alone, solving 18 problems in C++ and 20 in Python on PSB2 at least once across experiments. To assess generalizability, we employ GPT-3.5 and Llama 3 on the PSB2 and HumanEval-X benchmarks. Although SEIDR with these models does not surpass current state-of-the-art methods on the Python benchmarks, the results on HumanEval-C++ are promising. SEIDR with Llama 3-8B achieves an average pass@100 of 84.2%. Across all SEIDR runs, 163 of 164 problems are solved at least once with GPT-3.5 in HumanEval-C++, and 162 of 164 with the smaller Llama 3-8B. We conclude that SEIDR effectively overcomes the near-miss syndrome in program synthesis with LLMs.

Förderung

  • European Commission through Horizon 2020
  • Research Council of Norway through the secureIT
  • Experimental Infrastructure for Exploration of Exascale Computing

Zitieren

APA

Grishina, A., Liventsev, V., Härmä, A., & Moonen, L. (2025). Fully Autonomous Programming Using Iterative Multi-Agent Debugging with Large Language Models. ACM Transactions on Evolutionary Learning and Optimization, 5(1), 1-37. https://doi.org/10.1145/3719351

Vancouver

Grishina A, Liventsev V, Härmä A, Moonen L. Fully Autonomous Programming Using Iterative Multi-Agent Debugging with Large Language Models. ACM Transactions on Evolutionary Learning and Optimization. 2025;5(1):1-37. doi:10.1145/3719351

BibTeX
@article{Grishina2025,
  author = {Grishina, Anastasiia and Liventsev, Vadim and Härmä, Aki and Moonen, Leon},
  title = {Fully Autonomous Programming Using Iterative Multi-Agent Debugging with Large Language Models},
  journal = {ACM Transactions on Evolutionary Learning and Optimization},
  year = {2025},
  volume = {5},
  number = {1},
  pages = {1--37},
  doi = {10.1145/3719351}
}
Kostenlos starten →