Using Confidence Modeling to Optimize Overall Score Quality in Hybrid Scoring Systems
Educational Measurement: Issues and Practice · 2026-06 · 45 (2)
DOI: 10.1111/emip.70019 ↗Kurz gefasst
Fachartikel in der Zeitschrift Educational Measurement: Issues and Practice, online erschienen am 05.04.2026. Verfasst von Alexander Kwako und 2 weiteren Autor:innen. Die Arbeit nennt 34 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
In large‐scale assessments, constructed response items are often scored using hybrid scoring systems, which combine human and automated scores. In this study, we augment automated scoring with confidence modeling to strategically route difficult‐to‐score responses for human review. We utilize hybrid performance curves to visualize the impact of routing on performance. Additionally, we propose several hybrid scoring policies for selecting optimal routing thresholds given practical constraints. Our findings reveal that hybrid scoring systems can achieve an overall performance that exceeds that of human‐ and automated‐only systems. Moreover, the superior performance of the hybrid system is less expensive than a human‐only system. These findings highlight the complementarity of human raters and automated scoring engines. Although current standards focus on the performance of human raters and automated scoring engines in isolation, we recommend that practitioners also report on the performance of the hybrid scoring system as a whole.
Zitieren
APA
Kwako, A., Lottridge, S., & Ormerod, C. (2026). Using Confidence Modeling to Optimize Overall Score Quality in Hybrid Scoring Systems. Educational Measurement: Issues and Practice, 45(2). https://doi.org/10.1111/emip.70019
Vancouver
Kwako A, Lottridge S, Ormerod C. Using Confidence Modeling to Optimize Overall Score Quality in Hybrid Scoring Systems. Educational Measurement: Issues and Practice. 2026;45(2). doi:10.1111/emip.70019
BibTeX
@article{Kwako2026,
author = {Kwako, Alexander and Lottridge, Susan and Ormerod, Christopher},
title = {Using Confidence Modeling to Optimize Overall Score Quality in Hybrid Scoring Systems},
journal = {Educational Measurement: Issues and Practice},
year = {2026},
volume = {45},
number = {2},
doi = {10.1111/emip.70019}
}
EUVIMED