EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Boosting LLMs for Mutation Generation

Bo Wang, Ming Deng, Mingda Chen, Chengran Yang, Youfang Lin, Mark Harman, Mike Papadakis, Jie M. Zhang

Proceedings of the ACM on Software Engineering · 30.06.2026 · 3 (FSE) · 3368-3391

DOI: 10.1145/3808156 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the ACM on Software Engineering, online erschienen am 30.06.2026. Verfasst von Bo Wang und 7 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 70 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Key Research and Development Program of China, National Natural Science Foundation of China, ITEA projects GreenCode.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

LLM-based mutation testing is a promising testing technology, but existing approaches typically rely on a fixed set of mutants as few-shot examples or none at all. This can result in generic low-quality mutants, missed context-specific mutation patterns, substantial numbers of redundant and non-compilable mutants, and limited semantic similarity to real bugs. To overcome these limitations, we introduce SMART (Semantic Mutation with Adaptive Retrieval and Tuning). SMART integrates retrieval-augmented generation (RAG) on a vectorized dataset of real-world bugs, focused code chunking, and supervised fine-tuning using mutants coupled with real-world bugs. We conducted an extensive empirical study of SMART using 1,991 real-world Java bugs from the Defects4J and ConDefects datasets, comparing SMART to the state-of-the-art LLM-based approaches, LLMut and LLMorpheus. The results reveal that SMART substantially improves mutation validity, effectiveness, and efficiency (even enabling 7B-scale models to match or even surpass large models like GPT-4o). We also demonstrate that SMART significantly improves downstream software engineering applications, including test case prioritization and fault localization. More specifically, SMART improves validity (weighted average generation rate) from 42.89% to 65.6%. It raises the non-duplicate rate from 87.38% to 95.62%, and the compilable rate from 88.85% to 90.21%. In terms of effectiveness, it achieves a real bug detection rate of 92.61% (vs. 57.86% for LLMut) and improves the average Ochiai coefficient from 25.61% to 38.44%. For fault localization, SMART ranks 64 more bugs as Top-1 under MUSE and 57 more under Metallaxis.

Förderung

  • National Key Research and Development Program of China
  • National Natural Science Foundation of China
  • ITEA projects GreenCode
  • GENIUS
  • Luxembourg National Research Fund (FNR) through the CORE project MiCE

Zitieren

APA

Wang, B., Deng, M., Chen, M., Yang, C., Lin, Y., Harman, M., Papadakis, M., & Zhang, J. M. (2026). Boosting LLMs for Mutation Generation. Proceedings of the ACM on Software Engineering, 3(FSE), 3368-3391. https://doi.org/10.1145/3808156

Vancouver

Wang B, Deng M, Chen M, Yang C, Lin Y, Harman M, et al. Boosting LLMs for Mutation Generation. Proceedings of the ACM on Software Engineering. 2026;3(FSE):3368-3391. doi:10.1145/3808156

BibTeX
@article{Wang2026,
  author = {Wang, Bo and Deng, Ming and Chen, Mingda and Yang, Chengran and Lin, Youfang and Harman, Mark and Papadakis, Mike and Zhang, Jie M.},
  title = {Boosting LLMs for Mutation Generation},
  journal = {Proceedings of the ACM on Software Engineering},
  year = {2026},
  volume = {3},
  number = {FSE},
  pages = {3368--3391},
  doi = {10.1145/3808156}
}
Kostenlos starten →