The Effects of Rating Designs on Rater Classification Accuracy and Rater Measurement Precision in Large-Scale Mixed-Format Assessments
Applied Psychological Measurement · 2023-03 · 47 (2) · 91-105
DOI: 10.1177/01466216231151705 ↗Kurz gefasst
Fachartikel in der Zeitschrift Applied Psychological Measurement, online erschienen am 12.01.2023. Verfasst von Wenjing Guo und 1 weiteren Autor:innen. Beteiligte Einrichtung: The University of Alabama, Tuscaloosa, AL, USA. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: Spencer Foundation.
Abstract (Originalsprache)
In standalone performance assessments, researchers have explored the influence of different rating designs on the sensitivity of latent trait model indicators to different rater effects as well as the impacts of different rating designs on student achievement estimates. However, the literature provides little guidance on the degree to which different rating designs might affect rater classification accuracy (severe/lenient) and rater measurement precision in both standalone performance assessments and mixed-format assessments. Using results from an analysis of National Assessment of Educational Progress (NAEP) data, we conducted simulation studies to systematically explore the impacts of different rating designs on rater measurement precision and rater classification accuracy (severe/lenient) in mixed-format assessments. The results suggest that the complete rating design produced the highest rater classification accuracy and greatest rater measurement precision, followed by the multiple-choice (MC) + spiral link design and the MC link design. Considering that complete rating designs are not practical in most testing situations, the MC + spiral link design may be a useful choice because it balances cost and performance. We consider the implications of our findings for research and practice.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- The University of Alabama, Tuscaloosa, AL, USA.
ORCID-Profile: Wenjing Guo, Stefanie A Wind
Förderung
- Spencer Foundation
Zitieren
APA
Guo, W., & Wind, S. A. (2023). The Effects of Rating Designs on Rater Classification Accuracy and Rater Measurement Precision in Large-Scale Mixed-Format Assessments. Applied Psychological Measurement, 47(2), 91-105. https://doi.org/10.1177/01466216231151705
Vancouver
Guo W, Wind SA. The Effects of Rating Designs on Rater Classification Accuracy and Rater Measurement Precision in Large-Scale Mixed-Format Assessments. Applied Psychological Measurement. 2023;47(2):91-105. doi:10.1177/01466216231151705
BibTeX
@article{Guo2023,
author = {Guo, Wenjing and Wind, Stefanie A},
title = {The Effects of Rating Designs on Rater Classification Accuracy and Rater Measurement Precision in Large-Scale Mixed-Format Assessments},
journal = {Applied Psychological Measurement},
year = {2023},
volume = {47},
number = {2},
pages = {91--105},
doi = {10.1177/01466216231151705}
}
EUVIMED