Quality Evaluation of Indonesian Student-generated User Stories: Insight from Human and ChatGPT Evaluation
International Journal of Modern Education and Computer Science · 08.04.2026 · 18 (2) · 82-99
DOI: 10.5815/ijmecs.2026.02.05 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Modern Education and Computer Science, erschienen am 08.04.2026. Verfasst von Muhammad Ihsan Zul und 5 weiteren Autor:innen.
Abstract (Originalsprache)
User stories are essential in agile software development for capturing software requirements, yet concerns over their quality persist globally. While prior studies have evaluated user story quality using practitioners and artificial intelligence, they primarily focus on general settings. This study addresses a gap by evaluating the quality of student-generated user stories in an educational context, specifically in Indonesia. The objective of this study is to compare evaluations by human evaluators and ChatGPT using the Quality User Story (QUS) Framework and evaluate the quality of the student-generated user story compared to the global studies. A total of 951 user stories from 103 student software projects were analyzed. Evaluations were conducted by three human evaluators and ChatGPT (GPT-4o). Percentage Agreement and Cohen’s Kappa measured inter-rater agreement, while the McNemar Test assessed statistical significance, and effect sizes were examined using Cohen’s g. Results show generally high agreement between human and ChatGPT evaluations, but lower consistency in several criteria, such as Conceptually Sound, Independent, and Unambiguous. Only four of the thirteen criteria—Conflict-Free, Unique, Well-Formed, and Atomic—showed no significant differences. Most criteria showed small to medium effect sizes, whereas Complete exhibited a large practical difference. Common quality issues among students included Uniform, Independent, and Complete (set criteria), Atomic, Conceptually Sound, and Unambiguous (individual criteria), with overlap observed in global studies. This study shows that ChatGPT can support user story evaluation in educational settings when guided by clear rubrics and validated by humans. It also offers practical insights for educators by identifying criteria that require stronger emphasis in teaching, particularly in software engineering education in Indonesia.
Zitieren
APA
Zul, M. I., Yasin, S. M., Chatisa, I., Imani, F. M., Helma, S. S., & Sahid, D. S. S. (2026). Quality Evaluation of Indonesian Student-generated User Stories: Insight from Human and ChatGPT Evaluation. International Journal of Modern Education and Computer Science, 18(2), 82-99. https://doi.org/10.5815/ijmecs.2026.02.05
Vancouver
Zul MI, Yasin SM, Chatisa I, Imani FM, Helma SS, Sahid DSS. Quality Evaluation of Indonesian Student-generated User Stories: Insight from Human and ChatGPT Evaluation. International Journal of Modern Education and Computer Science. 2026;18(2):82-99. doi:10.5815/ijmecs.2026.02.05
BibTeX
@article{Zul2026,
author = {Zul, Muhammad Ihsan and Yasin, Suhaila Mohd. and Chatisa, Ivan and Imani, Fikri Muhaffizh and Helma, Siti Syahidatul and Sahid, Dadang Syarif Sihabudin},
title = {Quality Evaluation of Indonesian Student-generated User Stories: Insight from Human and ChatGPT Evaluation},
journal = {International Journal of Modern Education and Computer Science},
year = {2026},
volume = {18},
number = {2},
pages = {82--99},
doi = {10.5815/ijmecs.2026.02.05}
}
EUVIMED