Evaluating the Consistency and Reliability of Attribution Methods in Automated Short Answer Grading (ASAG) Systems: Toward an Explainable Scoring System
Journal of Educational Measurement · 2025-06 · 62 (2) · 248-281
DOI: 10.1111/jedm.12438 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Educational Measurement, 2025-06. Verfasst von Wallace N. Pinto und 1 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 64 Literaturangaben.
Abstract (Originalsprache)
In recent years, the application of explainability techniques to automated essay scoring and automated short‐answer grading (ASAG) models, particularly those based on transformer architectures, has gained significant attention. However, the reliability and consistency of these techniques remain underexplored. This study systematically investigates the use of attribution scores in ASAG systems, focusing on their consistency in reflecting model decisions. Specifically, we examined how attribution scores generated by different methods—namely Local Interpretable Model‐agnostic Explanations (LIME), Integrated Gradients (IG), Hierarchical Explanation via Divisive Generation (HEDGE), and Leave‐One‐Out (LOO)—compare in their consistency and ability to illustrate the decision‐making processes of transformer‐based scoring systems trained on a publicly available response dataset. Additionally, we analyzed how attribution scores varied across different scoring categories in a polytomously scored response dataset and across two transformer‐based scoring model architectures: Bidirectional Encoder Representations from Transformers (BERT) and Decoding‐enhanced BERT with Disentangled Attention (DeBERTa‐v2). Our findings highlight the challenges in evaluating explainability metrics, with important implications for both high‐stakes and formative assessment contexts. This study contributes to the development of more reliable and transparent ASAG systems.
Zitieren
APA
Pinto, W. N., & Shin, J. (2025). Evaluating the Consistency and Reliability of Attribution Methods in Automated Short Answer Grading (ASAG) Systems: Toward an Explainable Scoring System. Journal of Educational Measurement, 62(2), 248-281. https://doi.org/10.1111/jedm.12438
Vancouver
Pinto WN, Shin J. Evaluating the Consistency and Reliability of Attribution Methods in Automated Short Answer Grading (ASAG) Systems: Toward an Explainable Scoring System. Journal of Educational Measurement. 2025;62(2):248-281. doi:10.1111/jedm.12438
BibTeX
@article{Pinto2025,
author = {Pinto, Wallace N. and Shin, Jinnie},
title = {Evaluating the Consistency and Reliability of Attribution Methods in Automated Short Answer Grading (ASAG) Systems: Toward an Explainable Scoring System},
journal = {Journal of Educational Measurement},
year = {2025},
volume = {62},
number = {2},
pages = {248--281},
doi = {10.1111/jedm.12438}
}
EUVIMED