EUVIMEDEUVIMED entdecken
Open Access

Evaluating AI-Based Automated Essay Scoring Through Signal Detection Theory: Beyond Aggregate Agreement Metrics

Xiaoliang Zhou iD

Applied Psychological Measurement · 23.07.2026

DOI: 10.1177/01466216261471171 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Psychological Measurement, online erschienen am 23.07.2026. Verfasst von Xiaoliang Zhou. Beteiligte Einrichtung: Methodology and Measurement, Australian Council for Educational Research, Melbourne, VIC, Australia. Die Arbeit nennt 103 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The rapid adoption of Large Language Models (LLMs) in educational assessment has reshaped scoring practices, yet evaluation remains tethered to aggregate reliability metrics like Quadratic Weighted Kappa, which obscure discrimination and rater effects. This study applies Signal Detection Theory to evaluate eight state-of-the-art LLMs (including Claude 3.5 Haiku, DeepSeek-V3, Gemini 3 Flash, GPT-4o, and Grok 4.1) against expert human raters across 1,726 essays. By decoupling discrimination from response criteria, I provide a diagnostic analysis of AI scoring behavior. Results indicate that human raters exhibit significantly superior evaluative precision, with average discrimination estimates approximately double those of the AI models. Furthermore, LLMs are prone to pronounced centrality effects and score compression, systematically failing to award the highest rubric tiers. These findings demonstrate that low human-machine agreement stems from both a deficit in discriminative accuracy and systematic shifts in response criteria. Ultimately, this research provides a robust framework for calibrating and selecting AI scoring systems based on specific pedagogical goals and fairness requirements.

Schlagwörter

Stichwörter der Autor:innen

automated essay scoringdiscriminationlarge language modelsresponse criteriasignal detection theory

Autor:innen und Institutionen

  • Methodology and Measurement, Australian Council for Educational Research, Melbourne, VIC, Australia.

ORCID-Profile: Xiaoliang Zhou

Zitieren

APA

Zhou, X. (2026). Evaluating AI-Based Automated Essay Scoring Through Signal Detection Theory: Beyond Aggregate Agreement Metrics. Applied Psychological Measurement. https://doi.org/10.1177/01466216261471171

Vancouver

Zhou X. Evaluating AI-Based Automated Essay Scoring Through Signal Detection Theory: Beyond Aggregate Agreement Metrics. Applied Psychological Measurement. 2026. doi:10.1177/01466216261471171

BibTeX
@article{Zhou2026,
  author = {Zhou, Xiaoliang},
  title = {Evaluating AI-Based Automated Essay Scoring Through Signal Detection Theory: Beyond Aggregate Agreement Metrics},
  journal = {Applied Psychological Measurement},
  year = {2026},
  doi = {10.1177/01466216261471171}
}
Kostenlos starten →