EUVIMEDEUVIMED entdecken
Crossref

A sequential approach to detecting differential rater functioning in sparse rater-mediated assessment networks

Stefanie A. Wind

Language Testing · 2023-04 · 40 (2) · 209-226

DOI: 10.1177/02655322221092388 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Language Testing, online erschienen am 12.05.2022. Verfasst von Stefanie A. Wind. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 37 Literaturangaben. Gefördert von: Spencer Foundation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Researchers frequently evaluate rater judgments in performance assessments for evidence of differential rater functioning (DRF), which occurs when rater severity is systematically related to construct-irrelevant student characteristics after controlling for student achievement levels. However, researchers have observed that methods for detecting DRF may be limited in sparse rating designs, where it is not possible for every rater to score every student. In these designs, there is limited information with which to detect DRF. Sparse designs can also exacerbate the impact of artificial DRF, which occurs when raters are inaccurately flagged for DRF due to statistical artifacts. In this study, a sequential method is adapted from previous research on differential item functioning (DIF) that allows researchers to detect DRF more accurately and distinguish between true and artificial DRF. Analyses of data from a rater-mediated writing assessment and a simulation study demonstrate that the sequential approach results in different conclusions about which raters exhibit DRF. Moreover, the simulation study results suggest that the sequential procedure results in improved accuracy in DRF detection across a variety of rating design conditions. Practical implications for language testing research are discussed.

Förderung

  • Spencer Foundation

Zitieren

APA

Wind, S. A. (2023). A sequential approach to detecting differential rater functioning in sparse rater-mediated assessment networks. Language Testing, 40(2), 209-226. https://doi.org/10.1177/02655322221092388

Vancouver

Wind SA. A sequential approach to detecting differential rater functioning in sparse rater-mediated assessment networks. Language Testing. 2023;40(2):209-226. doi:10.1177/02655322221092388

BibTeX
@article{Wind2023,
  author = {Wind, Stefanie A.},
  title = {A sequential approach to detecting differential rater functioning in sparse rater-mediated assessment networks},
  journal = {Language Testing},
  year = {2023},
  volume = {40},
  number = {2},
  pages = {209--226},
  doi = {10.1177/02655322221092388}
}
Kostenlos starten →