EXECUTION-FEEDBACK-GUIDED CONTEXT SELECTION FOR LARGE LANGUAGE MODEL-BASED SOFTWARE TESTING AND DEBUGGING
World Journal of Information and Knowledge Management · 24.06.2023 · 1 (1) · 55-64
DOI: 10.61784/wjikm4029 ↗Kurz gefasst
Fachartikel in der Zeitschrift World Journal of Information and Knowledge Management, online erschienen am 24.06.2023. Verfasst von TianJun Mo und 3 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Large language models (LLMs) are now routinely used to generate unit tests, localize faults and repair programs, yet their effectiveness on real projects hinges on a step that is rarely engineered explicitly: deciding which code enters the prompt. Whole files exceed context budgets, lexical retrieval misses the helper functions in which faults hide, and stack traces are silent for assertion failures. This paper presents EFCS, an execution-feedback-guided context selection approach that turns the by-products of test execution—test outcomes, exception traces, statement-level spectra and dynamic call edges—into a relevance model over function-level code units. EFCS fuses spectrum and traceback evidence, assembles a token-budgeted context from execution-trimmed function bodies and interface skeletons of runtime callees, and re-selects the context across debugging iterations through an evidence memory and a cost-aware feedback-escalation policy. We evaluate EFCS on 897 single-fault and 210 two-fault instances seeded into seven real-world Python projects with 4,182 developer-written tests, counting tokens with the cl100k tokenizer of GPT-3.5 and GPT-4. On held-out faults, EFCS places the faulty statement inside a 1K-token context for 94.6% of the faults, compared with 38.2% for BM25 retrieval, 58.7% for stack-trace selection and 9.6% for focal-file prompting; under a 2K budget it delivers interface-complete contexts for 95.3% of the faults while using 10.9% fewer tokens than spectrum-based selection. In a replay of iterative debugging, EFCS resolves 97.2% of two-fault instances within five model calls, lowers the share of wasted calls to 8.6%, and its escalation policy reduces traced test executions by 64.8% at a comparable resolution rate.
Zitieren
APA
Mo, T., Wang, G., Feng, E., & Li, P. (2023). EXECUTION-FEEDBACK-GUIDED CONTEXT SELECTION FOR LARGE LANGUAGE MODEL-BASED SOFTWARE TESTING AND DEBUGGING. World Journal of Information and Knowledge Management, 1(1), 55-64. https://doi.org/10.61784/wjikm4029
Vancouver
Mo T, Wang G, Feng E, Li P. EXECUTION-FEEDBACK-GUIDED CONTEXT SELECTION FOR LARGE LANGUAGE MODEL-BASED SOFTWARE TESTING AND DEBUGGING. World Journal of Information and Knowledge Management. 2023;1(1):55-64. doi:10.61784/wjikm4029
BibTeX
@article{Mo2023,
author = {Mo, TianJun and Wang, GaiKe and Feng, EnMiao and Li, PengFei},
title = {EXECUTION-FEEDBACK-GUIDED CONTEXT SELECTION FOR LARGE LANGUAGE MODEL-BASED SOFTWARE TESTING AND DEBUGGING},
journal = {World Journal of Information and Knowledge Management},
year = {2023},
volume = {1},
number = {1},
pages = {55--64},
doi = {10.61784/wjikm4029}
}
EUVIMED