EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

ProjectEvalPlus: An Agentic Software Engineering Benchmark with Automatic Language Extension and User Simulated Evaluation

Kaiyuan Liu, Youcheng Pan, Yexing Du, Lei Zhang, Daojing He, Yang Xiang

ACM Transactions on Software Engineering and Methodology · 30.11.2026 · 35 (11) · 1-19

DOI: 10.1145/3817119 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Software Engineering and Methodology, online erschienen am 04.11.2026. Verfasst von Kaiyuan Liu und 5 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 29 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Key R&D Program of China, National Science and Technology Major Program, National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large Language Models (LLMs) have shown impressive capabilities in Software Engineering (SWE) , yet evaluating their performance on realistic project-level SWE remains challenging. Existing benchmarks focus on isolated snippets, lacking project-level context, language extensibilities, and end-user reality. Thus, we introduce ProjectEvalPlus , a comprehensive user-interaction-simulation benchmark that extends automatic evaluation to multiple programming languages via Automatic Language Extensibility (ALE) , supports multi-agent scenario stages evaluation, and provides detailed feedback on logical and runtime errors. Through unified testing on Python, Java, and JavaScript, we demonstrate that ProjectEvalPlus effectively assesses LLM agents’ capabilities, reveals language-specific biases, and enables iterative cross-language adaptation. Our results highlight both the strengths and limitations of current LLM agents in practical software development, offering actionable insights for future multi-language, project-level software engineering evaluation.

Förderung

  • National Key R&D Program of China
  • National Science and Technology Major Program
  • National Natural Science Foundation of China
  • Guangdong S&T Program
  • Shenzhen Science and Technology Program
  • Major Key Project of PCL

Zitieren

APA

Liu, K., Pan, Y., Du, Y., Zhang, L., He, D., & Xiang, Y. (2026). ProjectEvalPlus: An Agentic Software Engineering Benchmark with Automatic Language Extension and User Simulated Evaluation. ACM Transactions on Software Engineering and Methodology, 35(11), 1-19. https://doi.org/10.1145/3817119

Vancouver

Liu K, Pan Y, Du Y, Zhang L, He D, Xiang Y. ProjectEvalPlus: An Agentic Software Engineering Benchmark with Automatic Language Extension and User Simulated Evaluation. ACM Transactions on Software Engineering and Methodology. 2026;35(11):1-19. doi:10.1145/3817119

BibTeX
@article{Liu2026,
  author = {Liu, Kaiyuan and Pan, Youcheng and Du, Yexing and Zhang, Lei and He, Daojing and Xiang, Yang},
  title = {ProjectEvalPlus: An Agentic Software Engineering Benchmark with Automatic Language Extension and User Simulated Evaluation},
  journal = {ACM Transactions on Software Engineering and Methodology},
  year = {2026},
  volume = {35},
  number = {11},
  pages = {1--19},
  doi = {10.1145/3817119}
}
Kostenlos starten →