Diagnosing the role of observable distribution shift in effect generalization for psychological experiments
Journal of the Royal Statistical Society Series C: Applied Statistics · 16.06.2026 · 75 (3) · 801-828
DOI: 10.1093/jrsssc/qlag019 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the Royal Statistical Society Series C: Applied Statistics, online erschienen am 15.05.2026. Verfasst von Ying Jin und 2 weiteren Autor:innen. Die Arbeit nennt 65 Literaturangaben. Gefördert von: Stanford University’s Human-Centered Artificial Intelligence.
Abstract (Originalsprache)
Social science researchers are making increasing efforts to replicate important experiments, compare effect estimates and analyse their discrepancies. This article is motivated by a replication of a psychological experiment in which a decline in the effect of an eye movement intervention on false memory has been attributed to a distributional shift in participants’ depression scores. Understanding how such shifts contribute to effect discrepancy is crucial for result reporting, scientific understanding, and treatment deployment. We propose a framework using generalizability methods to decompose effect size discrepancy between two studies into contributions of various sources of distribution shifts. We address several unique challenges in the application, including incorporating commonly cited sources of shift within a unified framework, providing interpretable summaries of contributions, dealing with limited sample sizes and selection bias. In the motivating study, our analysis reveals that, despite the notable shift in observed depression scores, due to limited effect heterogeneity, their contribution to effect discrepancy is less compelling than that of unobserved factors. In two additional pairs of experiments, our method offers context-dependent insights on the contributions of different factors in effect generalization. The proposed tools are especially useful early in the scientific process when there are not enough studies for a meta-analysis.
Förderung
- Stanford University’s Human-Centered Artificial Intelligence
Zitieren
APA
Jin, Y., Guo, K., & Rothenhäusler, D. (2026). Diagnosing the role of observable distribution shift in effect generalization for psychological experiments. Journal of the Royal Statistical Society Series C: Applied Statistics, 75(3), 801-828. https://doi.org/10.1093/jrsssc/qlag019
Vancouver
Jin Y, Guo K, Rothenhäusler D. Diagnosing the role of observable distribution shift in effect generalization for psychological experiments. Journal of the Royal Statistical Society Series C: Applied Statistics. 2026;75(3):801-828. doi:10.1093/jrsssc/qlag019
BibTeX
@article{Jin2026,
author = {Jin, Ying and Guo, Kevin and Rothenhäusler, Dominik},
title = {Diagnosing the role of observable distribution shift in effect generalization for psychological experiments},
journal = {Journal of the Royal Statistical Society Series C: Applied Statistics},
year = {2026},
volume = {75},
number = {3},
pages = {801--828},
doi = {10.1093/jrsssc/qlag019}
}
EUVIMED