A Teacher-Centered Evaluation of AI-Assisted Educational Assessment: Rubric Consistency, Feedback Quality, and Fairness Across Student Groups
Academic Journal of Management and Social Sciences · 18.11.2026 · 18 (2) · 221-234
DOI: 10.54097/g70yrm22 ↗Kurz gefasst
Fachartikel in der Zeitschrift Academic Journal of Management and Social Sciences, online erschienen am 18.11.2026. Verfasst von Qianyi Fang. Die Arbeit nennt 29 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
AI-assisted assessment is increasingly used to score student writing and to generate formative feedback, yet it is still validated mainly by aggregate agreement with consensus scores. For classroom adoption the decisive question is teacher-centered: does the AI apply the rubric the way a competent teacher does, diagnose the same weaknesses, and treat every student group alike? We propose a Teacher-Centered Evaluation (TCE) framework with four lenses—teacher-relative rubric consistency, rubric-guided counterfactual tests, feedback diagnostic validity, and score-conditional fairness—complemented by a teacher-in-the-loop routing policy with group-audited conformal intervals. TCE is instantiated on two public corpora released by the research team behind the ASAP 2.0 benchmark: PERSUADE 2.0 (5,531 argumentative essays with holistic scores, 55,068 expert-rated discourse elements, and student demographics) and ELLIPSE (6,462 essays double-scored by 27 trained raters on seven rubric traits). Five AI scoring engines are audited. The strongest engine reaches a quadratic weighted kappa (QWK) of 0.870 and, in a teacher-substitution test, agrees with a teacher almost exactly as well as a second teacher does (ΔQWK = −0.016). TCE nevertheless exposes failures that agreement statistics hide: scores are blind to sentence order, rise by 0.27 points when a paragraph is duplicated, and fall no more when evidence is deleted than when equally long non-evidence text is removed; analytic trait scores collapse into a single dimension (inter-trait r = 0.87 vs. 0.61 for teachers); feedback misses 70% of the weaknesses that experts flag in English-learner writing; and conditioning on the teacher’s score uncovers subgroup gaps that raw statistics mask. Routing 20% of essays to teachers with a learned deferral model raises QWK to 0.907, reduces the worst subgroup bias by 31%, and nearly halves the English-learner agreement gap.
Zitieren
APA
Fang, Q. (2026). A Teacher-Centered Evaluation of AI-Assisted Educational Assessment: Rubric Consistency, Feedback Quality, and Fairness Across Student Groups. Academic Journal of Management and Social Sciences, 18(2), 221-234. https://doi.org/10.54097/g70yrm22
Vancouver
Fang Q. A Teacher-Centered Evaluation of AI-Assisted Educational Assessment: Rubric Consistency, Feedback Quality, and Fairness Across Student Groups. Academic Journal of Management and Social Sciences. 2026;18(2):221-234. doi:10.54097/g70yrm22
BibTeX
@article{Fang2026,
author = {Fang, Qianyi},
title = {A Teacher-Centered Evaluation of AI-Assisted Educational Assessment: Rubric Consistency, Feedback Quality, and Fairness Across Student Groups},
journal = {Academic Journal of Management and Social Sciences},
year = {2026},
volume = {18},
number = {2},
pages = {221--234},
doi = {10.54097/g70yrm22}
}
EUVIMED