Boosting LLMs for Mutation Generation
Proceedings of the ACM on Software Engineering · 30.06.2026 · 3 (FSE) · 3368-3391
DOI: 10.1145/3808156 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the ACM on Software Engineering, online erschienen am 30.06.2026. Verfasst von Bo Wang und 7 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 70 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Key Research and Development Program of China, National Natural Science Foundation of China, ITEA projects GreenCode.
Abstract (Originalsprache)
LLM-based mutation testing is a promising testing technology, but existing approaches typically rely on a fixed set of mutants as few-shot examples or none at all. This can result in generic low-quality mutants, missed context-specific mutation patterns, substantial numbers of redundant and non-compilable mutants, and limited semantic similarity to real bugs. To overcome these limitations, we introduce SMART (Semantic Mutation with Adaptive Retrieval and Tuning). SMART integrates retrieval-augmented generation (RAG) on a vectorized dataset of real-world bugs, focused code chunking, and supervised fine-tuning using mutants coupled with real-world bugs. We conducted an extensive empirical study of SMART using 1,991 real-world Java bugs from the Defects4J and ConDefects datasets, comparing SMART to the state-of-the-art LLM-based approaches, LLMut and LLMorpheus. The results reveal that SMART substantially improves mutation validity, effectiveness, and efficiency (even enabling 7B-scale models to match or even surpass large models like GPT-4o). We also demonstrate that SMART significantly improves downstream software engineering applications, including test case prioritization and fault localization. More specifically, SMART improves validity (weighted average generation rate) from 42.89% to 65.6%. It raises the non-duplicate rate from 87.38% to 95.62%, and the compilable rate from 88.85% to 90.21%. In terms of effectiveness, it achieves a real bug detection rate of 92.61% (vs. 57.86% for LLMut) and improves the average Ochiai coefficient from 25.61% to 38.44%. For fault localization, SMART ranks 64 more bugs as Top-1 under MUSE and 57 more under Metallaxis.
Förderung
- National Key Research and Development Program of China
- National Natural Science Foundation of China
- ITEA projects GreenCode
- GENIUS
- Luxembourg National Research Fund (FNR) through the CORE project MiCE
Zitieren
APA
Wang, B., Deng, M., Chen, M., Yang, C., Lin, Y., Harman, M., Papadakis, M., & Zhang, J. M. (2026). Boosting LLMs for Mutation Generation. Proceedings of the ACM on Software Engineering, 3(FSE), 3368-3391. https://doi.org/10.1145/3808156
Vancouver
Wang B, Deng M, Chen M, Yang C, Lin Y, Harman M, et al. Boosting LLMs for Mutation Generation. Proceedings of the ACM on Software Engineering. 2026;3(FSE):3368-3391. doi:10.1145/3808156
BibTeX
@article{Wang2026,
author = {Wang, Bo and Deng, Ming and Chen, Mingda and Yang, Chengran and Lin, Youfang and Harman, Mark and Papadakis, Mike and Zhang, Jie M.},
title = {Boosting LLMs for Mutation Generation},
journal = {Proceedings of the ACM on Software Engineering},
year = {2026},
volume = {3},
number = {FSE},
pages = {3368--3391},
doi = {10.1145/3808156}
}
EUVIMED