EUVIMEDEUVIMED entdecken
Crossref

Item-Level Heterogeneity in Value Added Models: Implications for Reliability, Cross-Study Comparability, and Effect Sizes

Joshua B. Gilbert, Zachary Himmelsbach, Luke W. Miratrix, Andrew D. Ho, Benjamin W. Domingue

Journal of Educational and Behavioral Statistics · 28.12.2025

DOI: 10.3102/10769986251393339 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Educational and Behavioral Statistics, online erschienen am 28.12.2025. Verfasst von Joshua B. Gilbert und 4 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 135 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Value added models (VAMs) attempt to estimate the causal effects of teachers and schools on student test scores. We apply Generalizability Theory to show how estimated VA effects depend upon the selection of test items. Standard VAMs estimate causal effects on the items that are included on the test. Generalizability demands consideration of how estimates would differ had the test included alternative items. We introduce a model that estimates the magnitude of item-by-teacher/school variance accurately, revealing that standard VAMs can overstate reliability and overestimate differences between units. Using 16 academic outcomes from 8 studies with item-level data, we show how standard VAMs overstate reliability by a median of 0.04 on the 0 to 1 reliability scale (mean = 0.09, SD = 0.10) and provide standard deviations of teacher/school effects that are a median of 3% too large (mean = 12%, SD = 23% points). We discuss how imprecision due to heterogeneous VA effects across items attenuates effect sizes, complicates comparisons across studies, and contributes to temporal instability, though these effects are reduced when the number of items is high. Our results suggest that accurate estimation and interpretation of VAMs may be improved using item-level data, including qualitative data about how items represent the content domain.

Zitieren

APA

Gilbert, J. B., Himmelsbach, Z., Miratrix, L. W., Ho, A. D., & Domingue, B. W. (2025). Item-Level Heterogeneity in Value Added Models: Implications for Reliability, Cross-Study Comparability, and Effect Sizes. Journal of Educational and Behavioral Statistics. https://doi.org/10.3102/10769986251393339

Vancouver

Gilbert JB, Himmelsbach Z, Miratrix LW, Ho AD, Domingue BW. Item-Level Heterogeneity in Value Added Models: Implications for Reliability, Cross-Study Comparability, and Effect Sizes. Journal of Educational and Behavioral Statistics. 2025. doi:10.3102/10769986251393339

BibTeX
@article{Gilbert2025,
  author = {Gilbert, Joshua B. and Himmelsbach, Zachary and Miratrix, Luke W. and Ho, Andrew D. and Domingue, Benjamin W.},
  title = {Item-Level Heterogeneity in Value Added Models: Implications for Reliability, Cross-Study Comparability, and Effect Sizes},
  journal = {Journal of Educational and Behavioral Statistics},
  year = {2025},
  doi = {10.3102/10769986251393339}
}
Kostenlos starten →