How Prompt Specificity Affects Edge Case Handling in LLM-Generated Code: An Empirical Evaluation
Artificial Intelligence and Machine Learning Review · 05.11.2024 · 5 (4) · 139-149
DOI: 10.69987/aimlr.2024.50411 ↗Kurz gefasst
Fachartikel in der Zeitschrift Artificial Intelligence and Machine Learning Review, online erschienen am 05.11.2024. Verfasst von Minhao Li und 2 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Large language models have demonstrated strong performance on code generation benchmarks, yet standard evaluations may overestimate their robustness by relying on insufficient test suites that fail to exercise edge cases. This study investigates how varying levels of prompt specificity influence the ability of LLMs to generate code that correctly handles edge cases. We define four incremental specificity levels ranging from minimal function signatures to prompts containing explicit edge case hints and evaluate four LLMs (GPT-4o, Claude 3.5 Sonnet, DeepSeek-Coder-V2, and Qwen2.5-Coder-32B) on the HumanEval+ benchmark, which augments 164 HumanEval problems with approximately 80 times more test cases targeting boundary conditions. We introduce the Edge Pass Rate (EPR) metric to isolate edge case handling from general functional correctness. Our results show that increasing prompt specificity from minimal to edge-explicit yields a mean EPR improvement of 15.9 percentage points across all models, roughly 1.8 times the corresponding pass@1 gain of 8.9 points. Boundary value and negative number categories benefit most, while type coercion edge cases remain resistant. Weaker models exhibit greater sensitivity to prompt specificity, suggesting that prompt investment yields disproportionate returns when computational resources constrain the choice of LLM.
Zitieren
APA
Li, M., Zhao, F., & Tang, T. (2024). How Prompt Specificity Affects Edge Case Handling in LLM-Generated Code: An Empirical Evaluation. Artificial Intelligence and Machine Learning Review, 5(4), 139-149. https://doi.org/10.69987/aimlr.2024.50411
Vancouver
Li M, Zhao F, Tang T. How Prompt Specificity Affects Edge Case Handling in LLM-Generated Code: An Empirical Evaluation. Artificial Intelligence and Machine Learning Review. 2024;5(4):139-149. doi:10.69987/aimlr.2024.50411
BibTeX
@article{Li2024,
author = {Li, Minhao and Zhao, Fanyi and Tang, Tianxing},
title = {How Prompt Specificity Affects Edge Case Handling in LLM-Generated Code: An Empirical Evaluation},
journal = {Artificial Intelligence and Machine Learning Review},
year = {2024},
volume = {5},
number = {4},
pages = {139--149},
doi = {10.69987/aimlr.2024.50411}
}
EUVIMED