EUVIMEDEUVIMED entdecken
Crossref

™ChatGPT for automated writing evaluation: Scoring and feedback across prompt conditions

Yewon Lee, Myunghwan Hwang

Language Testing · 25.03.2026

DOI: 10.1177/02655322261425759 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Language Testing, online erschienen am 25.03.2026. Verfasst von Yewon Lee und 1 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 65 Literaturangaben. Gefördert von: Soonchunhyang University.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This study investigates ChatGPT’s performance as an Automated Writing Evaluation (AWE) system by comparing its scoring with that of human raters and examining learners’ perceptions of its feedback. Six ChatGPT models were developed using different prompt configurations. Sixty English writing samples produced by Korean university English as a Foreign Language (EFL) learners were evaluated by two human raters and the six models. A multifaceted Rasch model, Spearman’s correlation, and intraclass correlation were used to examine reliability, severity, and bias. Learners’ perspectives on the models’ feedback were collected through open-ended surveys and analyzed thematically. The results indicate that prompt design plays a central role in shaping ChatGPT’s scoring behavior. Prompts combining Chain-of-Thought reasoning with Fill-in-the-blank scaffolding were associated with higher scoring consistency, while predefined personas and few-shot exemplars tended to moderate scoring severity. However, no stable patterns were observed for either bias or rating scale use, suggesting that prompt design alone cannot fully control domain-level bias. In particular, reasoning-intensive writing domains showed substantial divergence from human judgment, highlighting the need for human oversight. In parallel, learners generally viewed ChatGPT’s feedback positively, while also noting areas for improvement. Overall, the study demonstrates the potential of prompt-calibrated ChatGPT-based AWE as a supplementary tool for writing assessment and instruction.

Förderung

  • Soonchunhyang University

Zitieren

APA

Lee, Y., & Hwang, M. (2026). ™ChatGPT for automated writing evaluation: Scoring and feedback across prompt conditions. Language Testing. https://doi.org/10.1177/02655322261425759

Vancouver

Lee Y, Hwang M. ™ChatGPT for automated writing evaluation: Scoring and feedback across prompt conditions. Language Testing. 2026. doi:10.1177/02655322261425759

BibTeX
@article{Lee2026,
  author = {Lee, Yewon and Hwang, Myunghwan},
  title = {™ChatGPT for automated writing evaluation: Scoring and feedback across prompt conditions},
  journal = {Language Testing},
  year = {2026},
  doi = {10.1177/02655322261425759}
}
Kostenlos starten →