INTEGRATED EVALUATION OF A SOCRATIC CONVERSATIONAL AGENT FOR INTRODUCTORY PROGRAMMING: A HUMAN-CENTERED DESIGN STUDY AT THE UNIVERSITY OF COSTA RICA
International Journal on Cybernetics & Informatics · 29.10.2026 · 15 (5) · 39-49
DOI: 10.5121/ijci.2026.150504 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal on Cybernetics & Informatics, online erschienen am 29.10.2026. Verfasst von Hansell Solís Ramírez und 1 weiteren Autor:innen.
Abstract (Originalsprache)
Large language models (MLLs) offer complete solutions for the programming field, thus representing a potential danger. Therefore, this article presents a user-centered Socratic conversational agent and its evaluation, focusing on introductory programming at the University of Costa Rica. Twenty students solved a timed Python task, while five evaluation sources were triangulated: perceived usability using the System Usability Scale (SUS), cognitive load using the NASA Task Load Index (NASA-TLX), interaction logs (411 prompts), a fidelity audit of system prompts (374 responses), and a focus group. Based on this data collection, a moderate-to-low usability score (SUS = 55.75, grade D) and a moderate cognitive load with a temporary peak were obtained, negatively correlated with usability (r = −0.68, 95% CI [−0.92, −0.09], p = 0.03). This represents a compromise in the interaction. On the other hand, the records showed a range of turns (from 3 to 87 per session), and it is worth noting that there were only seven direct requests for a solution. Another important point is the agent's adherence to the imposed rules, revealing a bias toward evaluative language in 35% of the responses. Meanwhile, the rule against using code was maintained in more than 96% of cases. A focus group identified a friction threshold of three to six exchanges and a distinction between declarative language and reasoning that defines when Socratic guidance is pedagogically appropriate. These findings will guide the basis for creating a redesign of the agent.
Zitieren
APA
Ramírez, H. S., & López, G. (2026). INTEGRATED EVALUATION OF A SOCRATIC CONVERSATIONAL AGENT FOR INTRODUCTORY PROGRAMMING: A HUMAN-CENTERED DESIGN STUDY AT THE UNIVERSITY OF COSTA RICA. International Journal on Cybernetics & Informatics, 15(5), 39-49. https://doi.org/10.5121/ijci.2026.150504
Vancouver
Ramírez HS, López G. INTEGRATED EVALUATION OF A SOCRATIC CONVERSATIONAL AGENT FOR INTRODUCTORY PROGRAMMING: A HUMAN-CENTERED DESIGN STUDY AT THE UNIVERSITY OF COSTA RICA. International Journal on Cybernetics & Informatics. 2026;15(5):39-49. doi:10.5121/ijci.2026.150504
BibTeX
@article{Ramrez2026,
author = {Ramírez, Hansell Solís and López, Gustavo},
title = {INTEGRATED EVALUATION OF A SOCRATIC CONVERSATIONAL AGENT FOR INTRODUCTORY PROGRAMMING: A HUMAN-CENTERED DESIGN STUDY AT THE UNIVERSITY OF COSTA RICA},
journal = {International Journal on Cybernetics & Informatics},
year = {2026},
volume = {15},
number = {5},
pages = {39--49},
doi = {10.5121/ijci.2026.150504}
}
EUVIMED