EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead

Junda He, Jieke Shi, Terry Yue Zhuo, Christoph Treude, Jiamou Sun, Xiaoning Du, Zhenchang Xing, David Lo

ACM Transactions on Software Engineering and Methodology · 30.09.2026 · 35 (9) · 1-30

DOI: 10.1145/3797276 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Software Engineering and Methodology, online erschienen am 21.08.2026. Verfasst von Junda He und 7 weiteren Autor:innen. Die Arbeit wurde bisher 19-mal zitiert (Crossref) und nennt 131 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Research Foundation Singapore, under its Investigatorship.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The rapid integration of Large Language Models (LLMs) into software engineering (SE) has revolutionized tasks from code generation to program repair, producing a massive volume of software artifacts. This surge in automated creation has exposed a critical bottleneck: the lack of scalable and reliable methods to evaluate the quality of these outputs. Human evaluation, while effective, is very costly and time-consuming. Traditional automated metrics like BLEU rely on high-quality references and struggle to capture nuanced aspects of software quality, such as readability and usefulness. In response, the LLM-as-a-Judge paradigm, which employs LLMs for automated evaluation, has emerged. This approach leverages the advanced reasoning and coding capabilities of LLMs themselves to perform automated evaluations, offering a compelling path toward achieving both the nuance of human insight and the scalability of automated systems. Nevertheless, LLM-as-a-Judge research in the SE community is still in its early stages, with many breakthroughs needed. This forward-looking SE 2030 paper aims to steer the research community toward advancing LLM-as-a-Judge for evaluating LLM-generated software artifacts, while also sharing potential research paths to achieve this goal. We provide a literature review of existing SE studies on LLM-as-a-Judge and envision these frameworks as reliable, robust, and scalable human surrogates capable of evaluating software artifacts with consistent, multi-faceted assessments by 2030 and beyond. To validate this vision, we analyze the limitations of current studies, identify key research gaps, and outline a detailed roadmap to guide future developments of LLM-as-a-Judge in software engineering. While not intended to be a definitive guide, our work aims to foster further research and adoption of LLM-as-a-Judge frameworks within the SE community, ultimately improving the effectiveness and scalability of software artifact evaluation methods.

Förderung

  • National Research Foundation Singapore, under its Investigatorship

Zitieren

APA

He, J., Shi, J., Zhuo, T. Y., Treude, C., Sun, J., Du, X., Xing, Z., & Lo, D. (2026). LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead. ACM Transactions on Software Engineering and Methodology, 35(9), 1-30. https://doi.org/10.1145/3797276

Vancouver

He J, Shi J, Zhuo TY, Treude C, Sun J, Du X, et al. LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead. ACM Transactions on Software Engineering and Methodology. 2026;35(9):1-30. doi:10.1145/3797276

BibTeX
@article{He2026,
  author = {He, Junda and Shi, Jieke and Zhuo, Terry Yue and Treude, Christoph and Sun, Jiamou and Du, Xiaoning and Xing, Zhenchang and Lo, David},
  title = {LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead},
  journal = {ACM Transactions on Software Engineering and Methodology},
  year = {2026},
  volume = {35},
  number = {9},
  pages = {1--30},
  doi = {10.1145/3797276}
}
Kostenlos starten →