Inter-rater agreement in Korean constructed-response writing assessment: a large-scale empirical baseline from the AI Hub 2024 writing evaluation dataset
Frontiers in Education · 23.07.2026 · 11
DOI: 10.3389/feduc.2026.1887497 ↗Kurz gefasst
Fachartikel in der Zeitschrift Frontiers in Education, online erschienen am 23.07.2026. Verfasst von Taeryeong Kim. Die Arbeit nennt 37 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Introduction
Inter-rater agreement is the principal quality criterion for large-scale human-annotated datasets used to train and benchmark automated essay scoring (AES) systems. Despite rapid growth in Korean AES research, no study has provided a comprehensive, criterion-level inter-rater agreement baseline for a publicly available Korean writing dataset. This study addresses that gap using the AI Hub 2024 Writing Automated Evaluation and Feedback Dataset, comprising 12,873 elementary fifth-grade responses across four subjects and two formats (short-answer, essay), scored by 42 raters on nine criteria, yielding 204,894 individual rater-item judgments.
Methods
We report Quadratic Weighted Kappa (QWK), Pearson r, and ICC(2,1) for each criterion, decompose agreement by format and subject via bootstrap inference, and apply Many-Facet Rasch Measurement (MFRM) to estimate and compare criterion-difficulty, rater-severity, and format/subject facet parameters. A secondary exploratory analysis examines whether an a priori coding of scoring-rule abstractness (rubric specificity) is associated with criterion-level QWK.
Results
Criterion-level QWK ranges from .432 to .924, with a 3:1 ratio of criterion-difficulty range to rater-severity range indicating that criterion-level variation structurally dominates rater-level variation. Rubric specificity is associated with inter-rater agreement (Spearman rho = −.686, p = .041, N = 9 criteria): rule-based criteria show near-ceiling agreement (mean QWK = .915) while judgment-based criteria show substantially lower agreement (mean QWK = .528). Given the small number of criteria, this association should be interpreted as descriptive rather than as a hypothesis test. Format effects are significant for six of nine criteria. The within-one agreement rate of 95.4% across all criteria provides a practical benchmark for evaluating AES models under a one-point tolerance criterion.
Discussion
These findings provide the measurement-theoretical infrastructure for designing rubric-specificity-stratified AES models, format-aware training protocols, and distributional scoring outputs for Korean constructed-response writing assessment.
Zitieren
APA
Kim, T. (2026). Inter-rater agreement in Korean constructed-response writing assessment: a large-scale empirical baseline from the AI Hub 2024 writing evaluation dataset. Frontiers in Education, 11. https://doi.org/10.3389/feduc.2026.1887497
Vancouver
Kim T. Inter-rater agreement in Korean constructed-response writing assessment: a large-scale empirical baseline from the AI Hub 2024 writing evaluation dataset. Frontiers in Education. 2026;11. doi:10.3389/feduc.2026.1887497
BibTeX
@article{Kim2026,
author = {Kim, Taeryeong},
title = {Inter-rater agreement in Korean constructed-response writing assessment: a large-scale empirical baseline from the AI Hub 2024 writing evaluation dataset},
journal = {Frontiers in Education},
year = {2026},
volume = {11},
doi = {10.3389/feduc.2026.1887497}
}
EUVIMED