How consistent is the algorithm? Examining the intra- and inter-rater reliability of LLM-based writing assessment
Frontiers in Education · 13.08.2026 · 11
DOI: 10.3389/feduc.2026.1861960 ↗Kurz gefasst
Fachartikel in der Zeitschrift Frontiers in Education, online erschienen am 13.08.2026. Verfasst von Bayan AlAmir. Die Arbeit nennt 39 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Unlike constrained constructed-response assessments, the reliability of essay-based assessments has long been debated in language education research. Recently, AI—particularly large language models—has been proposed as a tool to enhance scoring reliability. This study examines the reliability of LLM-based scoring in writing assessment. OpenAI's ChatGPT assessed 192 essays written by EFL learners using a criterion-based analytic rubric covering task achievement, grammatical range and accuracy, lexical resources, organization, and mechanics. Scoring was replicated after three weeks under similar conditions. The two rounds were compared to assess intra-rater reliability and then contrasted with human ratings to assess inter-rater reliability. Results indicated high intra-rater reliability, particularly for task achievement and organization. Lower consistency was observed for mechanics and grammatical accuracy, suggesting a need for clearer criteria or further model training. Human–AI comparisons showed slight to fair exact agreement (Cohen's kappa) and moderate agreement when accounting for the degree of disagreement.
Zitieren
APA
AlAmir, B. (2026). How consistent is the algorithm? Examining the intra- and inter-rater reliability of LLM-based writing assessment. Frontiers in Education, 11. https://doi.org/10.3389/feduc.2026.1861960
Vancouver
AlAmir B. How consistent is the algorithm? Examining the intra- and inter-rater reliability of LLM-based writing assessment. Frontiers in Education. 2026;11. doi:10.3389/feduc.2026.1861960
BibTeX
@article{AlAmir2026,
author = {AlAmir, Bayan},
title = {How consistent is the algorithm? Examining the intra- and inter-rater reliability of LLM-based writing assessment},
journal = {Frontiers in Education},
year = {2026},
volume = {11},
doi = {10.3389/feduc.2026.1861960}
}
EUVIMED