Automated Essay Scoring With GPT ‐4 for a Local Placement Test: Investigating Prompting Strategies, Intra‐Rater Reliability, and Alignment With Human Scores
TESOL Quarterly · 2025-12 · 59 (S1)
DOI: 10.1002/tesq.3405 ↗Kurz gefasst
Fachartikel in der Zeitschrift TESOL Quarterly, online erschienen am 25.05.2025. Verfasst von Yoonseo Kim. Die Arbeit wurde bisher 23-mal zitiert (Crossref) und nennt 14 Literaturangaben.
Abstract (Originalsprache)
This study explores the potential of OpenAI's ChatGPT‐4 (gpt‐4‐0613) as an automated essay scoring (AES) tool in a trial involving 300 essays from an American university's academic English program placement test. Three prompting strategies (minimal/detailed rubric, require/not require rationale, and with/without scoring examples) were tested for intra‐rater reliability and agreement with human ratings and placements. The study also examined improvements through NLP‐derived linguistic features and averaging multiple GPT‐4 ratings. Results revealed promising indicators such as high intra‐rater reliability, moderate positive correlations with human scores, and better placement agreement than the human average. Prompting strategies influenced scoring accuracy. A prompting strategy combining the provision of a detailed rubric, a request for a scoring rationale, and the provision of scoring examples demonstrated the best performance. Adding linguistic features and averaging multiple GPT‐4 ratings were even more effective. The findings suggest that while relying entirely on GPT‐4 for essay scoring might be premature, incorporating it as a complement to human ratings could be beneficial.
Zitieren
APA
Kim, Y. (2025). Automated Essay Scoring With GPT ‐4 for a Local Placement Test: Investigating Prompting Strategies, Intra‐Rater Reliability, and Alignment With Human Scores. TESOL Quarterly, 59(S1). https://doi.org/10.1002/tesq.3405
Vancouver
Kim Y. Automated Essay Scoring With GPT ‐4 for a Local Placement Test: Investigating Prompting Strategies, Intra‐Rater Reliability, and Alignment With Human Scores. TESOL Quarterly. 2025;59(S1). doi:10.1002/tesq.3405
BibTeX
@article{Kim2025,
author = {Kim, Yoonseo},
title = {Automated Essay Scoring With GPT ‐4 for a Local Placement Test: Investigating Prompting Strategies, Intra‐Rater Reliability, and Alignment With Human Scores},
journal = {TESOL Quarterly},
year = {2025},
volume = {59},
number = {S1},
doi = {10.1002/tesq.3405}
}
EUVIMED