EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

A Comparative Evaluation of Prompting Strategies for Code Generation with Large Language Models

Fanyi Zhao, Mingzhuo Yu, Chuankai Luo

Journal of Global Engineering Review · 06.01.2024 · 2 (1) · 1-11

DOI: 10.66372/jger.v2i1.1 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Global Engineering Review, online erschienen am 06.01.2024. Verfasst von Fanyi Zhao und 2 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models have demonstrated remarkable capabilities in automated code generation, yet the relative effectiveness of different prompting strategies remains insufficiently characterized under zcontrolled experimental conditions. This study presents a systematic comparative evaluation of six prompting strategies — zero-shot, few-shot, chain-of-thought, structured chain-of-thought, self-debugging, and self-refine — across four representative models (GPT-4o, GPT-3.5-Turbo, DeepSeek-Coder-33B-Instruct, and CodeLlama-34B-Instruct) on three established benchmarks (HumanEval+, MBPP+, and LiveCodeBench). The evaluation encompasses functional correctness measured by pass@1, computational cost quantified by average token consumption, and difficulty-stratified performance analysis on contamination-free competitive programming tasks. Results indicate that structured chain-of-thought prompting yields the most consistent improvements across all models, achieving 6.7–8.5 percentage point gains over zero-shot baselines on HumanEval+. Multi-turn strategies such as self-debugging deliver the highest absolute performance on frontier models (80.5% pass@1 for GPT-4o on HumanEval+) while offering diminished returns on smaller open-source models. Cost-effectiveness analysis reveals that single-turn structured prompting achieves 77–93% of multi-turn performance at 72% lower token cost. These findings provide empirically grounded guidance for practitioners selecting prompting strategies under varying model capability levels and computational resource constraints.

Zitieren

APA

Zhao, F., Yu, M., & Luo, C. (2024). A Comparative Evaluation of Prompting Strategies for Code Generation with Large Language Models. Journal of Global Engineering Review, 2(1), 1-11. https://doi.org/10.66372/jger.v2i1.1

Vancouver

Zhao F, Yu M, Luo C. A Comparative Evaluation of Prompting Strategies for Code Generation with Large Language Models. Journal of Global Engineering Review. 2024;2(1):1-11. doi:10.66372/jger.v2i1.1

BibTeX
@article{Zhao2024,
  author = {Zhao, Fanyi and Yu, Mingzhuo and Luo, Chuankai},
  title = {A Comparative Evaluation of Prompting Strategies for Code Generation with Large Language Models},
  journal = {Journal of Global Engineering Review},
  year = {2024},
  volume = {2},
  number = {1},
  pages = {1--11},
  doi = {10.66372/jger.v2i1.1}
}
Kostenlos starten →