EUVIMEDEUVIMED entdecken
Crossref

A Comprehensive Study on Large Language Models for Mutation Testing

Bo Wang, Mingda Chen, Ming Deng, Youfang Lin, Mark Harman, Mike Papadakis, Jie M. Zhang

ACM Transactions on Software Engineering and Methodology · 28.03.2026

DOI: 10.1145/3805038 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Software Engineering and Methodology, online erschienen am 28.03.2026. Verfasst von Bo Wang und 6 weiteren Autor:innen. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 98 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large Language Models (LLMs) have recently been used to generate mutants both in the research community and in industrial practice. However, there has been no comprehensive empirical study of their performance for this increasingly important LLM-based Software Engineering application. To address this, we conduct a comprehensive empirical study evaluating both a wide range of traditional approaches and LLM-based approaches. Particularly, we evaluate BugFarm, LLMorpheus, and our newly designed prompt for mutant generation. The experiments cover both leading open- and closed-source LLMs, on 851 real bugs from two Java real-world bug benchmarks. Our results reveal that, compared to existing traditional approaches, LLMs generate more diverse mutants that are behaviorally closer to real bugs and, most importantly, achieve a 1.8 \(\times\) improvement in real bug detection, defined as the proportion of real bugs whose faulty behaviors can be mimicked by at least one generated mutant. Specifically, LLM-based approaches reach a detection rate of 77.4%, compared to 41.6% for rule-based techniques, representing an absolute gain of 35.8 percentage points. Nevertheless, our results also reveal that these impressive improvements in effectiveness come at a cost: LLM-generated mutants exhibit worse non-compilability, duplication, and equivalent mutant rates than rule-based approaches by 25.9, 7.1, and 2.6 percentage points, respectively. These findings provide actionable insights for both research and practice. They allow practitioners to have greater confidence in deploying LLM-based mutation, while researchers now have a baseline for the state-of-the-art, with which they can research techniques to further improve effectiveness and reduce cost.

Zitieren

APA

Wang, B., Chen, M., Deng, M., Lin, Y., Harman, M., Papadakis, M., & Zhang, J. M. (2026). A Comprehensive Study on Large Language Models for Mutation Testing. ACM Transactions on Software Engineering and Methodology. https://doi.org/10.1145/3805038

Vancouver

Wang B, Chen M, Deng M, Lin Y, Harman M, Papadakis M, et al. A Comprehensive Study on Large Language Models for Mutation Testing. ACM Transactions on Software Engineering and Methodology. 2026. doi:10.1145/3805038

BibTeX
@article{Wang2026,
  author = {Wang, Bo and Chen, Mingda and Deng, Ming and Lin, Youfang and Harman, Mark and Papadakis, Mike and Zhang, Jie M.},
  title = {A Comprehensive Study on Large Language Models for Mutation Testing},
  journal = {ACM Transactions on Software Engineering and Methodology},
  year = {2026},
  doi = {10.1145/3805038}
}
Kostenlos starten →