Understanding Student Effectiveness with AI Chatbots in Introductory Programming Education: An Interaction Analysis Study
Proceedings of the Canadian Engineering Education Association (CEEA) · 07.08.2026
DOI: 10.24908/pceea.2026.21646 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the Canadian Engineering Education Association (CEEA), online erschienen am 07.08.2026. Verfasst von Rubaina Khan und 4 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
Generative AI chatbots are increasingly used in introductory programming courses, but whether they support learning or encourage over-reliance remains unclear. To examine which interaction features are associated with productive persistence, we analyzed 198 CS1 student–chatbot conversation logs using a five-dimensional coding scheme: query type, response relevance, response type, dialogue outcome, and effectiveness marker. Persistence was measured as the number of student prompts per log. Negative binomial models showed that relevance failures predicted longer interactions, particularly false positives (IRR = 2.49) and false negatives (IRR = 1.65), both p < 0.001. Logistic models showed that false negatives strongly predicted unresolved sessions (OR = 9.94, p = 0.008), suggesting that the inability to answer accelerates abandonment. In contrast, conceptual (Socratic) responses predicted progressive effectiveness markers. Overall, interaction quality, not mere usage, appears to drive productive persistence, highlighting the importance of minimizing false negatives and encouraging Socratic scaffolding in educational chatbot design and deployment.
Zitieren
APA
Khan, R., Siderius, J., Ross, K. J., Sari, Y., & Frank, B. (2026). Understanding Student Effectiveness with AI Chatbots in Introductory Programming Education: An Interaction Analysis Study. Proceedings of the Canadian Engineering Education Association (CEEA). https://doi.org/10.24908/pceea.2026.21646
Vancouver
Khan R, Siderius J, Ross KJ, Sari Y, Frank B. Understanding Student Effectiveness with AI Chatbots in Introductory Programming Education: An Interaction Analysis Study. Proceedings of the Canadian Engineering Education Association (CEEA). 2026. doi:10.24908/pceea.2026.21646
BibTeX
@article{Khan2026,
author = {Khan, Rubaina and Siderius, Joshua and Ross, Kyle James and Sari, Yuksel and Frank, Brian},
title = {Understanding Student Effectiveness with AI Chatbots in Introductory Programming Education: An Interaction Analysis Study},
journal = {Proceedings of the Canadian Engineering Education Association (CEEA)},
year = {2026},
doi = {10.24908/pceea.2026.21646}
}
EUVIMED