EUVIMEDEUVIMED entdecken
Open Access

The Effects of Rating Designs on Rater Classification Accuracy and Rater Measurement Precision in Large-Scale Mixed-Format Assessments

Wenjing Guo iD, Stefanie A Wind iD

Applied Psychological Measurement · 2023-03 · 47 (2) · 91-105

DOI: 10.1177/01466216231151705 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Psychological Measurement, online erschienen am 12.01.2023. Verfasst von Wenjing Guo und 1 weiteren Autor:innen. Beteiligte Einrichtung: The University of Alabama, Tuscaloosa, AL, USA. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: Spencer Foundation.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In standalone performance assessments, researchers have explored the influence of different rating designs on the sensitivity of latent trait model indicators to different rater effects as well as the impacts of different rating designs on student achievement estimates. However, the literature provides little guidance on the degree to which different rating designs might affect rater classification accuracy (severe/lenient) and rater measurement precision in both standalone performance assessments and mixed-format assessments. Using results from an analysis of National Assessment of Educational Progress (NAEP) data, we conducted simulation studies to systematically explore the impacts of different rating designs on rater measurement precision and rater classification accuracy (severe/lenient) in mixed-format assessments. The results suggest that the complete rating design produced the highest rater classification accuracy and greatest rater measurement precision, followed by the multiple-choice (MC) + spiral link design and the MC link design. Considering that complete rating designs are not practical in most testing situations, the MC + spiral link design may be a useful choice because it balances cost and performance. We consider the implications of our findings for research and practice.

Schlagwörter

Stichwörter der Autor:innen

NAEPmixed-format assessmentsrating designsrating quality

Autor:innen und Institutionen

  • The University of Alabama, Tuscaloosa, AL, USA.

ORCID-Profile: Wenjing Guo, Stefanie A Wind

Förderung

  • Spencer Foundation

Zitieren

APA

Guo, W., & Wind, S. A. (2023). The Effects of Rating Designs on Rater Classification Accuracy and Rater Measurement Precision in Large-Scale Mixed-Format Assessments. Applied Psychological Measurement, 47(2), 91-105. https://doi.org/10.1177/01466216231151705

Vancouver

Guo W, Wind SA. The Effects of Rating Designs on Rater Classification Accuracy and Rater Measurement Precision in Large-Scale Mixed-Format Assessments. Applied Psychological Measurement. 2023;47(2):91-105. doi:10.1177/01466216231151705

BibTeX
@article{Guo2023,
  author = {Guo, Wenjing and Wind, Stefanie A},
  title = {The Effects of Rating Designs on Rater Classification Accuracy and Rater Measurement Precision in Large-Scale Mixed-Format Assessments},
  journal = {Applied Psychological Measurement},
  year = {2023},
  volume = {47},
  number = {2},
  pages = {91--105},
  doi = {10.1177/01466216231151705}
}
Kostenlos starten →