Empirische Arbeit: One Model to Score Them All? On Suitability, Stability, and Synergy in Automated Essay Evaluation
Psychologie in Erziehung und Unterricht · 01.07.2026 · 73 (3)
DOI: 10.2378/peu2026.art13d ↗Kurz gefasst
Fachartikel in der Zeitschrift Psychologie in Erziehung und Unterricht, online erschienen am 01.07.2026. Verfasst von Andrea Horbach und 5 weiteren Autor:innen.
Abstract (Originalsprache)
Writing is a key educational competence whose development depends on feedback. Automated Essay Scoring (AES) is increasingly used to support feedback generation, yet most prior research has neglected score stability across repeated runs. However, inconsistent scoring can undermine trust in AES and limit its educational value. We evaluate feature-based logistic regression models, transformer-based neural models, and generative large language models on 4,593 EFL essays from the MEWS dataset. Beyond accuracy, we analyze prediction variability across multiple runs and examine agreement patterns within and across model families. Results show that feature-based models remain competitive, while LLMs achieve high accuracy. Despite strong average performance, GPT-5 exhibits substantial variability across runs. Across models, agreement patterns reveal that different families succeed and fail on different item subsets. Our findings underline stability as a crucial dimension for deploying AES models in educational contexts and highlight the need for careful model selection and potentially model combination.
Zitieren
APA
Horbach, A., Melanchthon, D. M., Schaller, N., Keller, S., Meyer, J., & Jansen, T. (2026). Empirische Arbeit: One Model to Score Them All? On Suitability, Stability, and Synergy in Automated Essay Evaluation. Psychologie in Erziehung und Unterricht, 73(3). https://doi.org/10.2378/peu2026.art13d
Vancouver
Horbach A, Melanchthon DM, Schaller N, Keller S, Meyer J, Jansen T. Empirische Arbeit: One Model to Score Them All? On Suitability, Stability, and Synergy in Automated Essay Evaluation. Psychologie in Erziehung und Unterricht. 2026;73(3). doi:10.2378/peu2026.art13d
BibTeX
@article{Horbach2026,
author = {Horbach, Andrea and Melanchthon, Daniel Mora and Schaller, Nils-Jonathan and Keller, Stefan and Meyer, Jennifer and Jansen, Thorben},
title = {Empirische Arbeit: One Model to Score Them All? On Suitability, Stability, and Synergy in Automated Essay Evaluation},
journal = {Psychologie in Erziehung und Unterricht},
year = {2026},
volume = {73},
number = {3},
doi = {10.2378/peu2026.art13d}
}
EUVIMED