EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Testing standards for AI-based scores in automated essay scoring

Rudolf Debelak iD, Matthias Ziegler iD

PLOS One · 31.07.2026 · 21 (7) · e0354680

DOI: 10.1371/journal.pone.0354680 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS One, online erschienen am 31.07.2026. Verfasst von Rudolf Debelak und 1 weiteren Autor:innen. Beteiligte Einrichtung: Institute of Education, University of Zurich, Zurich, Switzerland. Hauptthemen laut MEDLINE: Artificial Intelligence, Educational Measurement. Die Arbeit nennt 75 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recent developments in the field of artificial intelligence and machine learning allow the wide application of large language models for the evaluation of written text and other non-numerical data. When applied in the context of psychological and educational assessments, such models can be used for assigning scores to essays and other types of responses. In contrast to classical tests, essays do not consist of test items, which leads to specific challenges in the evaluation of testing standards for scores obtained from AI models that differ from those observed for classical ability tests and personality questionnaires. To address these challenges, we discuss the evaluation of validity, fairness, and reliability for scores obtained from models of artificial intelligence in the context of automated essay scoring. We review existing methods, propose new methods, and further illustrate the reviewed methods with an empirical example based on the Hewlett Foundation data set on automated essay scoring. By applying the proposed framework to an evaluation based on a DistilBERT model, we find the model to be robust with sufficiently high internal consistency (Spearman-Brown coefficients in the range from .77 to .92). We further found empirical evidence for the validity of the evaluation model, but also indications for violations of fairness when comparing the human and AI scores across different topics. This study provides a standardized, replicable toolkit for researchers and practitioners to evaluate the psychometric quality of AI-based assessments.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceEducational MeasurementHumansReproducibility of ResultsLarge Language ModelsMachine Learning

Autor:innen und Institutionen

  • Institute of Education, University of Zurich, Zurich, Switzerland.
  • MLO Lab, EPFL, Lausanne, Vaud, Switzerland.
  • Department of Psychology, Humboldt Universität zu Berlin, Berlin, Germany.

ORCID-Profile: Rudolf Debelak, Matthias Ziegler

Zitieren

APA

Debelak, R., & Ziegler, M. (2026). Testing standards for AI-based scores in automated essay scoring. PLOS One, 21(7), e0354680. https://doi.org/10.1371/journal.pone.0354680

Vancouver

Debelak R, Ziegler M. Testing standards for AI-based scores in automated essay scoring. PLOS One. 2026;21(7):e0354680. doi:10.1371/journal.pone.0354680

BibTeX
@article{Debelak2026,
  author = {Debelak, Rudolf and Ziegler, Matthias},
  title = {Testing standards for AI-based scores in automated essay scoring},
  journal = {PLOS One},
  year = {2026},
  volume = {21},
  number = {7},
  pages = {e0354680},
  doi = {10.1371/journal.pone.0354680}
}
Kostenlos starten →