EUVIMEDEUVIMED entdecken
Crossref

Diagnosing the role of observable distribution shift in effect generalization for psychological experiments

Ying Jin, Kevin Guo, Dominik Rothenhäusler

Journal of the Royal Statistical Society Series C: Applied Statistics · 16.06.2026 · 75 (3) · 801-828

DOI: 10.1093/jrsssc/qlag019 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the Royal Statistical Society Series C: Applied Statistics, online erschienen am 15.05.2026. Verfasst von Ying Jin und 2 weiteren Autor:innen. Die Arbeit nennt 65 Literaturangaben. Gefördert von: Stanford University’s Human-Centered Artificial Intelligence.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Social science researchers are making increasing efforts to replicate important experiments, compare effect estimates and analyse their discrepancies. This article is motivated by a replication of a psychological experiment in which a decline in the effect of an eye movement intervention on false memory has been attributed to a distributional shift in participants’ depression scores. Understanding how such shifts contribute to effect discrepancy is crucial for result reporting, scientific understanding, and treatment deployment. We propose a framework using generalizability methods to decompose effect size discrepancy between two studies into contributions of various sources of distribution shifts. We address several unique challenges in the application, including incorporating commonly cited sources of shift within a unified framework, providing interpretable summaries of contributions, dealing with limited sample sizes and selection bias. In the motivating study, our analysis reveals that, despite the notable shift in observed depression scores, due to limited effect heterogeneity, their contribution to effect discrepancy is less compelling than that of unobserved factors. In two additional pairs of experiments, our method offers context-dependent insights on the contributions of different factors in effect generalization. The proposed tools are especially useful early in the scientific process when there are not enough studies for a meta-analysis.

Förderung

  • Stanford University’s Human-Centered Artificial Intelligence

Zitieren

APA

Jin, Y., Guo, K., & Rothenhäusler, D. (2026). Diagnosing the role of observable distribution shift in effect generalization for psychological experiments. Journal of the Royal Statistical Society Series C: Applied Statistics, 75(3), 801-828. https://doi.org/10.1093/jrsssc/qlag019

Vancouver

Jin Y, Guo K, Rothenhäusler D. Diagnosing the role of observable distribution shift in effect generalization for psychological experiments. Journal of the Royal Statistical Society Series C: Applied Statistics. 2026;75(3):801-828. doi:10.1093/jrsssc/qlag019

BibTeX
@article{Jin2026,
  author = {Jin, Ying and Guo, Kevin and Rothenhäusler, Dominik},
  title = {Diagnosing the role of observable distribution shift in effect generalization for psychological experiments},
  journal = {Journal of the Royal Statistical Society Series C: Applied Statistics},
  year = {2026},
  volume = {75},
  number = {3},
  pages = {801--828},
  doi = {10.1093/jrsssc/qlag019}
}
Kostenlos starten →