EUVIMEDEUVIMED entdecken
Crossref

Evaluating Rater Effects of Large Language Models in Automated Essay Scoring: GPT, Claude, Gemini, and DeepSeek

Hong Jiao, Dan Song, Won‐Chan Lee

Educational Measurement: Issues and Practice · 2026-06 · 45 (2)

DOI: 10.1111/emip.70018 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Educational Measurement: Issues and Practice, online erschienen am 12.03.2026. Verfasst von Hong Jiao und 2 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 66 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) have been widely explored for automated scoring in educational assessment to facilitate learning and instruction. However, empirical evidence regarding which LLMs produce the most reliable scores and induce the least rater effects remains limited. This study compared 10 LLMs (ChatGPT 3.5, ChatGPT 4, ChatGPT 4o, OpenAI o1, Claude 3.5 Sonnet, Gemini 1.5, Gemini 1.5 Pro, Gemini 2.0, DeepSeek V3, and DeepSeek R1) with human expert raters in scoring two types of writing tasks. Their performance was evaluated in terms of score accuracy, intra‐rater consistency, and rater effects estimated using the Many‐Facet Rasch model. Although the results generally supported the use of ChatGPT 4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet with high scoring accuracy, better intra‐rater consistency, and less rater effects, the study is not intended to support substantive comparisons or rankings of LLMs or to identify a single “best” model, given the small sample size.

Zitieren

APA

Jiao, H., Song, D., & Lee, W. (2026). Evaluating Rater Effects of Large Language Models in Automated Essay Scoring: GPT, Claude, Gemini, and DeepSeek. Educational Measurement: Issues and Practice, 45(2). https://doi.org/10.1111/emip.70018

Vancouver

Jiao H, Song D, Lee W. Evaluating Rater Effects of Large Language Models in Automated Essay Scoring: GPT, Claude, Gemini, and DeepSeek. Educational Measurement: Issues and Practice. 2026;45(2). doi:10.1111/emip.70018

BibTeX
@article{Jiao2026,
  author = {Jiao, Hong and Song, Dan and Lee, Won‐Chan},
  title = {Evaluating Rater Effects of Large Language Models in Automated Essay Scoring: GPT, Claude, Gemini, and DeepSeek},
  journal = {Educational Measurement: Issues and Practice},
  year = {2026},
  volume = {45},
  number = {2},
  doi = {10.1111/emip.70018}
}
Kostenlos starten →