EUVIMEDEUVIMED entdecken
Crossref

Revisiting reliability with human and machine learning raters under scoring design and rater configuration in the many‐facet Rasch model

Xingyao Xiao iD, Richard J Patz, Mark R Wilson

British Journal of Mathematical and Statistical Psychology · 2026-11 · 79 (3) · 764-784

DOI: 10.1111/bmsp.70034 ↗

Kurz gefasst

Fachartikel in der Zeitschrift British Journal of Mathematical and Statistical Psychology, online erschienen am 31.01.2026. Verfasst von Xingyao Xiao und 2 weiteren Autor:innen. Beteiligte Einrichtung: Graduate School of Education, Stanford University, Stanford, California, USA. Hauptthemen laut MEDLINE: Psychometrics, Machine Learning, Models, Statistical. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 28 Literaturangaben. Gefördert von: National Science Foundation.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Constructed‐response (CR) items are widely used to assess higher order skills but require human scoring, which introduces variability and is costly at scale. Machine learning (ML)‐based scoring offers a scalable alternative, yet its psychometric consequences in rater‐mediated models remain underexplored. This study examines how scoring design, rater bias, ML inconsistency and model specification affect the reliability of ability estimation in polytomous CR assessments. Using Monte Carlo simulation, we manipulated human and ML rater bias, ML inconsistency and scoring density (complete, overlapping, isolated). Five estimation models were compared, including the Partial Credit Model (PCM) with fixed thresholds and the Many‐Facet Partial Credit Model (MFPCM) with and without free calibration. Results showed that systematic bias, not random inconsistency, was the main source of error. Hybrid human–ML scoring improved estimation when raters were unbiased or exhibited opposing biases, but error compounded when biases aligned. Across designs, PCM with fixed thresholds consistently outperformed more complex alternatives, while anchoring CR items to selected‐response metrics stabilized MFPCM estimation. The real data application replicated these patterns. Findings show that scoring design and bias structure, rather than model complexity, drive the benefits of hybrid scoring and that anchoring offers a practical strategy for stabilizing estimation.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

PsychometricsMachine LearningModels, StatisticalHumansReproducibility of ResultsMonte Carlo MethodComputer SimulationObserver VariationBias

Stichwörter der Autor:innen

Many‐Facet Partial Credit Model (MFPCM)Many‐Facet Rasch Model (MFRM)constructed‐response scoringmachine learning scoringrater configurationreliabilityscoring matrix design

Autor:innen und Institutionen

  • Graduate School of Education, Stanford University, Stanford, California, USA.
  • National Council on Measurement in Education, Mount Royal, New Jersey, USA.
  • Berkeley School of Education, University of California, Berkeley, Berkeley, California, USA.

ORCID-Profile: Xingyao Xiao

Förderung

  • National Science Foundation

Zitieren

APA

Xiao, X., Patz, R. J., & Wilson, M. R. (2026). Revisiting reliability with human and machine learning raters under scoring design and rater configuration in the many‐facet Rasch model. British Journal of Mathematical and Statistical Psychology, 79(3), 764-784. https://doi.org/10.1111/bmsp.70034

Vancouver

Xiao X, Patz RJ, Wilson MR. Revisiting reliability with human and machine learning raters under scoring design and rater configuration in the many‐facet Rasch model. British Journal of Mathematical and Statistical Psychology. 2026;79(3):764-784. doi:10.1111/bmsp.70034

BibTeX
@article{Xiao2026,
  author = {Xiao, Xingyao and Patz, Richard J and Wilson, Mark R},
  title = {Revisiting reliability with human and machine learning raters under scoring design and rater configuration in the many‐facet Rasch model},
  journal = {British Journal of Mathematical and Statistical Psychology},
  year = {2026},
  volume = {79},
  number = {3},
  pages = {764--784},
  doi = {10.1111/bmsp.70034}
}
Kostenlos starten →