Limitations of QWK in Evaluating Automated and Human Scoring Systems
Educational Measurement: Issues and Practice · 2026-03 · 45 (1)
DOI: 10.1111/emip.70017 ↗Kurz gefasst
Fachartikel in der Zeitschrift Educational Measurement: Issues and Practice, online erschienen am 25.02.2026. Verfasst von Jennifer Lewis und 1 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 37 Literaturangaben.
Abstract (Originalsprache)
To assess the interrater reliability of human ratings of constructed responses (CR), or the accuracy of scores given by automated scoring engines, concordance metrics quantify agreement between measures. This article examines the quadratic weighted kappa (QWK) in these contexts and highlights its practical limitations compared to other metrics. Both empirical and simulation study results reveal how different factors including the shape of the marginal distributions and score scale length may impact the estimates and how we can adjust for these properties of the contingency table. The results highlight the QWK's sensitivities and suggest that additional caution should be taken before decisions about whether to keep a CR item on a test form are made. If using QWK without the proper interpretive supports, such decisions may be misinformed. Consequently, we make suggestions for best practices to promote responsible evaluation of agreement in the context of CR scoring in educational testing.
Zitieren
APA
Lewis, J., & Casabianca, J. M. (2026). Limitations of QWK in Evaluating Automated and Human Scoring Systems. Educational Measurement: Issues and Practice, 45(1). https://doi.org/10.1111/emip.70017
Vancouver
Lewis J, Casabianca JM. Limitations of QWK in Evaluating Automated and Human Scoring Systems. Educational Measurement: Issues and Practice. 2026;45(1). doi:10.1111/emip.70017
BibTeX
@article{Lewis2026,
author = {Lewis, Jennifer and Casabianca, Jodi M.},
title = {Limitations of QWK in Evaluating Automated and Human Scoring Systems},
journal = {Educational Measurement: Issues and Practice},
year = {2026},
volume = {45},
number = {1},
doi = {10.1111/emip.70017}
}
EUVIMED