EUVIMEDEUVIMED entdecken
Open Access · CC BY-ND 4.0

A study on the reliability of ChatGPT-based English writing assessment: Internal and external perspectives

Boyu Wang, Yunjia Zhang

Language Education & Assessment · 19.08.2026 · 9 · 103386

DOI: 10.29140/lea.2026.103386 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Language Education & Assessment, online erschienen am 19.08.2026. Verfasst von Boyu Wang und 1 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY-ND 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

With the advancement of big data and artificial intelligence, natural language processing (NLP) has been increasingly integrated into educational assessment, facilitating a shift from human to automated scoring in English writing assessment. This study investigates how prompt design, guided by the TELeR taxonomy (Santu & Feng, 2023), and sample training affects ChatGPT-4's scoring reliability. A sample of 120 IELTS-style essays was scored across four analytic dimensions--Task Response (TR), Coherence and Cohesion (CC), Lexical Resource (LR), and Grammatical Range and Accuracy (GRA)--as well as holistically. Internal (intra-rater) reliability was measured via ICC(3,1) across three scoring rounds, while external (inter-rater) reliability was assessed by integrating ChatGPT-4 as an additional rater into a panel of nine human scorers using ICC(2,1). The results revealed that ChatGPT-4's internal consistency was moderate and was markedly improved by detailed prompts on content-related dimensions (TR/CC), but showed minimal gains on linguistic dimensions (LR/GRA). Training with human-scored exemplars under the better prompt reduced discrepancies with human ratings for external reliability, yet slightly decreased internal consistency for most dimensions. These findings position TELeR as a framework linking prompt engineering to psychometric theory and underscore the value of dimension-specific prompt design for reliable AI-based writing assessment.

Zitieren

APA

Wang, B., & Zhang, Y. (2026). A study on the reliability of ChatGPT-based English writing assessment: Internal and external perspectives. Language Education & Assessment, 9, 103386. https://doi.org/10.29140/lea.2026.103386

Vancouver

Wang B, Zhang Y. A study on the reliability of ChatGPT-based English writing assessment: Internal and external perspectives. Language Education & Assessment. 2026;9:103386. doi:10.29140/lea.2026.103386

BibTeX
@article{Wang2026,
  author = {Wang, Boyu and Zhang, Yunjia},
  title = {A study on the reliability of ChatGPT-based English writing assessment: Internal and external perspectives},
  journal = {Language Education & Assessment},
  year = {2026},
  volume = {9},
  pages = {103386},
  doi = {10.29140/lea.2026.103386}
}
Kostenlos starten →