EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Model-Agnostic Empirical Evaluation of Test-Driven Prompt Engineering on Improving Accuracy and Efficiency in Large Language Models Python Code Generation

Muhammad Rizqullah, Emad Albassam

IEEE Access · 2026 · 14 · 22801-22821

DOI: 10.1109/access.2026.3662817 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Access, 2026. Verfasst von Muhammad Rizqullah und 1 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 48 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Deanship of Scientific Research (DSR), King Abdulaziz University, Jeddah, Saudi Arabia.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Although Large Language Models (LLMs) are widely adopted for Python code generation, the generated code can be semantically incorrect, requiring iterations of evaluation and refinement. Test-driven prompt (TDP) engineering systematically incorporates test cases within the prompt structure for code generation task. However, existing research lacks a comprehensive evaluation of TDP engineering across diverse LLMs, programming problems, and associated explainability analysis. In this paper, we propose a specification clarification mechanism to explain TDP’s effectiveness in generating Python code while also uncovering novel insights through multimodel analysis. To this end, we created an experimental framework that facilitates evaluation of TDP engineering against normal prompting across first-attempt and remediation scenarios using eight models (GPT-4o, GPT-4o-mini, Claude 3.5 Sonnet/Haiku, Qwen 32B/14B/7B/3B Coder) and three benchmarks (HumanEval, MBPP, Code Contests). Qualitative analysis of code produced with TDP for programming problems of different difficulty shows consistent accuracy gains, from following formatting specifications to offering safeguards against logical errors, where each level of difficulty shows its own main advantage. Quantitatively, TDP achieves superior aggregate performance in all of 16 model-dataset combinations, with 6.09% average improvement (95% CI: [4.01, 8.18], p ¡ 0.0001, Cohen’s d = 1.08). A key practical finding is that TDP enables democratization of high-quality code generation: smaller, cost-effective models using TDP consistently match or exceed the performance of larger, more expensive models. TDP achieves up to 20% overall computational efficiency in multi-attempt scenarios, especially for coding-specialized models. While this research focuses on Python, we present theoretical frameworks for future cross-language validation. As a result, TDP is a robust technique for enhancing code generation quality. Finally, to facilitate reproducibility of our results and further research, our experimental framework is made publicly accessible for researchers and practitioners.

Förderung

  • Deanship of Scientific Research (DSR), King Abdulaziz University, Jeddah, Saudi Arabia

Zitieren

APA

Rizqullah, M., & Albassam, E. (2026). Model-Agnostic Empirical Evaluation of Test-Driven Prompt Engineering on Improving Accuracy and Efficiency in Large Language Models Python Code Generation. IEEE Access, 14, 22801-22821. https://doi.org/10.1109/access.2026.3662817

Vancouver

Rizqullah M, Albassam E. Model-Agnostic Empirical Evaluation of Test-Driven Prompt Engineering on Improving Accuracy and Efficiency in Large Language Models Python Code Generation. IEEE Access. 2026;14:22801-22821. doi:10.1109/access.2026.3662817

BibTeX
@article{Rizqullah2026,
  author = {Rizqullah, Muhammad and Albassam, Emad},
  title = {Model-Agnostic Empirical Evaluation of Test-Driven Prompt Engineering on Improving Accuracy and Efficiency in Large Language Models Python Code Generation},
  journal = {IEEE Access},
  year = {2026},
  volume = {14},
  pages = {22801--22821},
  doi = {10.1109/access.2026.3662817}
}
Kostenlos starten →