EUVIMEDEUVIMED entdecken
Crossref

Measuring the Accuracy of True Score Predictions for AI Scoring Evaluation

Daniel F. McCaffrey, Jodi M. Casabianca, Matthew S. Johnson

Journal of Educational Measurement · 2025-12 · 62 (4) · 763-786

DOI: 10.1111/jedm.70011 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Educational Measurement, online erschienen am 12.10.2025. Verfasst von Daniel F. McCaffrey und 2 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 38 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Use of artificial intelligence (AI) to score responses is growing in popularity and likely to increase. Evidence of the validity of scores relies on quadratic weighted kappa (QWK) to demonstrate agreement between AI scores and human ratings. QWK is a measure of agreement that accounts for chance agreement and the ordinality of the data by giving greater weight to larger disagreements. It has known shortcomings including sensitivity to the human rating reliability. The proportional reduction in mean squared error (PRMSE) measures agreement between predictions and their target that accounts for measurement error in the target. For example, the accuracy of the automated scoring model, with respect to prediction of the human true scores rather than the observed ratings. Extensive simulation study results show PRMSE is robust to many factors to which QWK is sensitive such as the human rater reliability, skew in the data and the number of score points. Analysis of operational test data demonstrates QWK and PRMSE can lead to different conclusions about AI scores. We investigate sample size requirements for accurate estimation of PRMSE in the context of AI scoring, although the results could apply more generally to measures with similar distributions as those tested in our study.

Zitieren

APA

McCaffrey, D. F., Casabianca, J. M., & Johnson, M. S. (2025). Measuring the Accuracy of True Score Predictions for AI Scoring Evaluation. Journal of Educational Measurement, 62(4), 763-786. https://doi.org/10.1111/jedm.70011

Vancouver

McCaffrey DF, Casabianca JM, Johnson MS. Measuring the Accuracy of True Score Predictions for AI Scoring Evaluation. Journal of Educational Measurement. 2025;62(4):763-786. doi:10.1111/jedm.70011

BibTeX
@article{McCaffrey2025,
  author = {McCaffrey, Daniel F. and Casabianca, Jodi M. and Johnson, Matthew S.},
  title = {Measuring the Accuracy of True Score Predictions for AI Scoring Evaluation},
  journal = {Journal of Educational Measurement},
  year = {2025},
  volume = {62},
  number = {4},
  pages = {763--786},
  doi = {10.1111/jedm.70011}
}
Kostenlos starten →