EUVIMEDEUVIMED entdecken
Crossref

A Framework for Evaluating Prompt Engineering Techniques for PHP Code Generation Using Open-Source Large Language Models

Rapheal Baja Alemoh, Bilkisu Larai Muhammad-Bello

Faculty of Natural and Applied Sciences Journal of Computing and Applications · 31.03.2026 · 3 (1) · 61-73

DOI: 10.63561/jca.v3i1.1204 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Faculty of Natural and Applied Sciences Journal of Computing and Applications, online erschienen am 31.03.2026. Verfasst von Rapheal Baja Alemoh und 1 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

As generative Artificial Intelligence (AI) systems become increasingly integrated into software development workflows in software engineering, there is a need for rigorous and reproducible evaluation of how prompt engineering techniques influence the quality, reliability, and efficiency of code generated by open-source large language models (LLMs). This paper presents a structured empirical evaluation framework to assess the effects of prompt techniques (zero-shot, few-shot, and chain-of-thought) on PHP code generation using three commonly adopted open-source models ( CodeLLaMA, Mistral, and StarCoder2). A reproducible experimental pipeline was developed to execute controlled prompt–model templates across multiple coding tasks (easy, medium and difficult) in software engineering. The framework automatically measures multiple performance metrics, including functional accuracy (pass@1 and pass@10), execution time, memory usage, lines of code, cyclomatic complexity, and coding standard violations. The results indicated that the most effective prompt strategy was the use of few-shot prompting on all models and then zero-shot that demonstrated improvement with repeated sampling (pass@10). Tasks with reasoning requirements benefited from chain-of-thought prompting in some aspects, which made the code more complex and lengthier. In the analysis of the effect of model variability for the consideration of this study, two-way Analysis of Variance (ANOVA) indicated that the underlying model architecture is still a significant factor in determining the performance differences. The execution time and memory usage were similar for all the strategies in prompting, while the code quality measures indicated structural differences in the code based on the prompt design. These results indicate that prompt engineering influences observable performance trends while adhering to the model’s architecture and task-level limitations, hence the need to consider the combination of prompt techniques, model capacity, task types, and evaluation methodology. THe assessment framework contributes to the development of standardized benchmarking tools in the PHP ecosystem, aside the contribution to the body of empirical research. This framework is a foundation for potential future extension to multiple metrics, reproducible assessment of open-source LLMs across programming languages, and task domains.

Zitieren

APA

Alemoh, R. B., & Muhammad-Bello, B. L. (2026). A Framework for Evaluating Prompt Engineering Techniques for PHP Code Generation Using Open-Source Large Language Models. Faculty of Natural and Applied Sciences Journal of Computing and Applications, 3(1), 61-73. https://doi.org/10.63561/jca.v3i1.1204

Vancouver

Alemoh RB, Muhammad-Bello BL. A Framework for Evaluating Prompt Engineering Techniques for PHP Code Generation Using Open-Source Large Language Models. Faculty of Natural and Applied Sciences Journal of Computing and Applications. 2026;3(1):61-73. doi:10.63561/jca.v3i1.1204

BibTeX
@article{Alemoh2026,
  author = {Alemoh, Rapheal Baja and Muhammad-Bello, Bilkisu Larai},
  title = {A Framework for Evaluating Prompt Engineering Techniques for PHP Code Generation Using Open-Source Large Language Models},
  journal = {Faculty of Natural and Applied Sciences Journal of Computing and Applications},
  year = {2026},
  volume = {3},
  number = {1},
  pages = {61--73},
  doi = {10.63561/jca.v3i1.1204}
}
Kostenlos starten →