EUVIMEDEUVIMED entdecken
Crossref

Code-Aware Prompting: A Study of Coverage-Guided Test Generation in Regression Setting using LLM

Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Murali Krishna Ramanathan, Baishakhi Ray

Proceedings of the ACM on Software Engineering · 12.07.2024 · 1 (FSE) · 951-971

DOI: 10.1145/3643769 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the ACM on Software Engineering, online erschienen am 12.07.2024. Verfasst von Gabriel Ryan und 6 weiteren Autor:innen. Die Arbeit wurde bisher 147-mal zitiert (Crossref) und nennt 43 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Testing plays a pivotal role in ensuring software quality, yet conventional Search Based Software Testing (SBST) methods often struggle with complex software units, achieving suboptimal test coverage. Recent work using large language models (LLMs) for test generation have focused on improving generation quality through optimizing the test generation context and correcting errors in model outputs, but use fixed prompting strategies that prompt the model to generate tests without additional guidance. As a result LLM-generated testsuites still suffer from low coverage. In this paper, we present SymPrompt, a code-aware prompting strategy for LLMs in test generation. SymPrompt’s approach is based on recent work that demonstrates LLMs can solve more complex logical problems when prompted to reason about the problem in a multi-step fashion. We apply this methodology to test generation by deconstructing the testsuite generation process into a multi-stage sequence, each of which is driven by a specific prompt aligned with the execution paths of the method under test, and exposing relevant type and dependency focal context to the model. Our approach enables pretrained LLMs to generate more complete test cases without any additional training. We implement SymPrompt using the TreeSitter parsing framework and evaluate on a benchmark challenging methods from open source Python projects. SymPrompt enhances correct test generations by a factor of 5 and bolsters relative coverage by 26% for CodeGen2. Notably, when applied to GPT-4, SymPrompt improves coverage by over 2× compared to baseline prompting strategies.

Zitieren

APA

Ryan, G., Jain, S., Shang, M., Wang, S., Ma, X., Ramanathan, M. K., & Ray, B. (2024). Code-Aware Prompting: A Study of Coverage-Guided Test Generation in Regression Setting using LLM. Proceedings of the ACM on Software Engineering, 1(FSE), 951-971. https://doi.org/10.1145/3643769

Vancouver

Ryan G, Jain S, Shang M, Wang S, Ma X, Ramanathan MK, et al. Code-Aware Prompting: A Study of Coverage-Guided Test Generation in Regression Setting using LLM. Proceedings of the ACM on Software Engineering. 2024;1(FSE):951-971. doi:10.1145/3643769

BibTeX
@article{Ryan2024,
  author = {Ryan, Gabriel and Jain, Siddhartha and Shang, Mingyue and Wang, Shiqi and Ma, Xiaofei and Ramanathan, Murali Krishna and Ray, Baishakhi},
  title = {Code-Aware Prompting: A Study of Coverage-Guided Test Generation in Regression Setting using LLM},
  journal = {Proceedings of the ACM on Software Engineering},
  year = {2024},
  volume = {1},
  number = {FSE},
  pages = {951--971},
  doi = {10.1145/3643769}
}
Kostenlos starten →