Comparative Evaluation of Large Language Models in Computer Programming Education
Proceedings of the Canadian Engineering Education Association (CEEA) · 07.08.2026
DOI: 10.24908/pceea.2026.21576 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the Canadian Engineering Education Association (CEEA), online erschienen am 07.08.2026. Verfasst von Melina Najimi und 2 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
Delivering high-quality formative feedback in large introductory programming courses remains challenging despite the widespread use of automated grading systems. Although large language models (LLMs) can generate natural-language explanations grounded in student code and test outcomes, their diagnostic reliability and pedagogical suitability require systematic evaluation. This study presents a comparative analysis of six LLMs for generating formative feedback on introductory Java programs containing predefined defects under controlled conditions. Model outputs were evaluated across key dimensions of feedback quality and performance. Results reveal substantial cross-model variation, particularly in multi-defect scenarios. More precise fault localization was associated with higher diagnostic accuracy and completeness, but also with more frequent instructional-constraint violations. No single model consistently outperformed others across all criteria, and response latency did not reliably correspond to diagnostic quality. These findings underscore the context-dependent nature of LLM-generated formative programming feedback.
Zitieren
APA
Najimi, M., Yazdani, S., & Ahmadzadeh, M. (2026). Comparative Evaluation of Large Language Models in Computer Programming Education. Proceedings of the Canadian Engineering Education Association (CEEA). https://doi.org/10.24908/pceea.2026.21576
Vancouver
Najimi M, Yazdani S, Ahmadzadeh M. Comparative Evaluation of Large Language Models in Computer Programming Education. Proceedings of the Canadian Engineering Education Association (CEEA). 2026. doi:10.24908/pceea.2026.21576
BibTeX
@article{Najimi2026,
author = {Najimi, Melina and Yazdani, Saba and Ahmadzadeh, Marzieh},
title = {Comparative Evaluation of Large Language Models in Computer Programming Education},
journal = {Proceedings of the Canadian Engineering Education Association (CEEA)},
year = {2026},
doi = {10.24908/pceea.2026.21576}
}
EUVIMED