EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

A Teacher-Centered Evaluation of AI-Assisted Educational Assessment: Rubric Consistency, Feedback Quality, and Fairness Across Student Groups

Qianyi Fang

Academic Journal of Management and Social Sciences · 18.11.2026 · 18 (2) · 221-234

DOI: 10.54097/g70yrm22 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Academic Journal of Management and Social Sciences, online erschienen am 18.11.2026. Verfasst von Qianyi Fang. Die Arbeit nennt 29 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

AI-assisted assessment is increasingly used to score student writing and to generate formative feedback, yet it is still validated mainly by aggregate agreement with consensus scores. For classroom adoption the decisive question is teacher-centered: does the AI apply the rubric the way a competent teacher does, diagnose the same weaknesses, and treat every student group alike? We propose a Teacher-Centered Evaluation (TCE) framework with four lenses—teacher-relative rubric consistency, rubric-guided counterfactual tests, feedback diagnostic validity, and score-conditional fairness—complemented by a teacher-in-the-loop routing policy with group-audited conformal intervals. TCE is instantiated on two public corpora released by the research team behind the ASAP 2.0 benchmark: PERSUADE 2.0 (5,531 argumentative essays with holistic scores, 55,068 expert-rated discourse elements, and student demographics) and ELLIPSE (6,462 essays double-scored by 27 trained raters on seven rubric traits). Five AI scoring engines are audited. The strongest engine reaches a quadratic weighted kappa (QWK) of 0.870 and, in a teacher-substitution test, agrees with a teacher almost exactly as well as a second teacher does (ΔQWK = −0.016). TCE nevertheless exposes failures that agreement statistics hide: scores are blind to sentence order, rise by 0.27 points when a paragraph is duplicated, and fall no more when evidence is deleted than when equally long non-evidence text is removed; analytic trait scores collapse into a single dimension (inter-trait r = 0.87 vs. 0.61 for teachers); feedback misses 70% of the weaknesses that experts flag in English-learner writing; and conditioning on the teacher’s score uncovers subgroup gaps that raw statistics mask. Routing 20% of essays to teachers with a learned deferral model raises QWK to 0.907, reduces the worst subgroup bias by 31%, and nearly halves the English-learner agreement gap.

Zitieren

APA

Fang, Q. (2026). A Teacher-Centered Evaluation of AI-Assisted Educational Assessment: Rubric Consistency, Feedback Quality, and Fairness Across Student Groups. Academic Journal of Management and Social Sciences, 18(2), 221-234. https://doi.org/10.54097/g70yrm22

Vancouver

Fang Q. A Teacher-Centered Evaluation of AI-Assisted Educational Assessment: Rubric Consistency, Feedback Quality, and Fairness Across Student Groups. Academic Journal of Management and Social Sciences. 2026;18(2):221-234. doi:10.54097/g70yrm22

BibTeX
@article{Fang2026,
  author = {Fang, Qianyi},
  title = {A Teacher-Centered Evaluation of AI-Assisted Educational Assessment: Rubric Consistency, Feedback Quality, and Fairness Across Student Groups},
  journal = {Academic Journal of Management and Social Sciences},
  year = {2026},
  volume = {18},
  number = {2},
  pages = {221--234},
  doi = {10.54097/g70yrm22}
}
Kostenlos starten →