EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Joint Item Response Models for Manual and Automatic Scores on Open-Ended Test Items

Daniel Bengs iD, Ulf Brefeld, Ulf Kroehne, Fabian Zehner

Psychometrika · 2025

DOI: 10.1017/psy.2025.10018 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Psychometrika, 2025. Verfasst von Daniel Bengs und 3 weiteren Autor:innen. Beteiligte Einrichtung: Leibniz Institute for Research and Information in Education, Frankfurt, Germany. Hauptthemen laut MEDLINE: Educational Measurement, Psychometrics, Models, Statistical. Die Arbeit nennt 41 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Test items using open-ended response formats can increase an instrument’s construct validity. However, traditionally, their application in educational testing requires human coders to score the responses. Manual scoring not only increases operational costs but also prohibits the use of evidence from open-ended items to inform routing decisions in adaptive designs. Using machine learning and natural language processing, automatic scoring provides classifiers that can instantly assign scores to text responses. Although optimized for agreement with manual scores, automatic scoring is not perfectly accurate and introduces an additional source of error into the response process, leading to a misspecification of the measurement model used with the manual score. We propose two joint models for manual and automatic scores of automatically scored open-ended items. Our models extend a given model from Item Response Theory for the manual scores by a component for the automatic scores, accounting for classification errors. The models were evaluated using data from the Programme for International Student Assessment (2012) and simulated data, demonstrating their capacity to mitigate the impact of classification errors on ability estimation compared to a baseline that disregards classification errors.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Educational MeasurementPsychometricsModels, StatisticalHumansMachine LearningNatural Language Processing

Stichwörter der Autor:innen

automatic scoringitem response modelinglarge-scale assessment

Autor:innen und Institutionen

  • Leibniz Institute for Research and Information in Education, Frankfurt, Germany.
  • Leuphana University Lüneburg, Lüneburg, Germany.
  • Centre for International Student Assessment.

ORCID-Profile: Daniel Bengs

Zitieren

APA

Bengs, D., Brefeld, U., Kroehne, U., & Zehner, F. (2025). Joint Item Response Models for Manual and Automatic Scores on Open-Ended Test Items. Psychometrika. https://doi.org/10.1017/psy.2025.10018

Vancouver

Bengs D, Brefeld U, Kroehne U, Zehner F. Joint Item Response Models for Manual and Automatic Scores on Open-Ended Test Items. Psychometrika. 2025. doi:10.1017/psy.2025.10018

BibTeX
@article{Bengs2025,
  author = {Bengs, Daniel and Brefeld, Ulf and Kroehne, Ulf and Zehner, Fabian},
  title = {Joint Item Response Models for Manual and Automatic Scores on Open-Ended Test Items},
  journal = {Psychometrika},
  year = {2025},
  doi = {10.1017/psy.2025.10018}
}
Kostenlos starten →