A multi-lab experimental assessment reveals that replicability can be improved by using empirical estimates of genotype-by-lab interaction
PLOS Biology · 01.05.2023 · 21 (5) · e3002082
DOI: 10.1371/journal.pbio.3002082 ↗Kurz gefasst
Fachartikel in der Zeitschrift PLOS Biology, online erschienen am 01.05.2023. Verfasst von Iman Jaljuli und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Statistics and Operations Research, Tel-Aviv University, Tel-Aviv, Israel. Hauptthemen laut MEDLINE: Laboratories, Research Design. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 40 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NIDA NIH HHS, United States - Israel Binational Science Foundation, US–Israel Binational Science Foundation—US National Science Foundation.
Abstract (Originalsprache)
The utility of mouse and rat studies critically depends on their replicability in other laboratories. A widely advocated approach to improving replicability is through the rigorous control of predefined animal or experimental conditions, known as standardization. However, this approach limits the generalizability of the findings to only to the standardized conditions and is a potential cause rather than solution to what has been called a replicability crisis. Alternative strategies include estimating the heterogeneity of effects across laboratories, either through designs that vary testing conditions, or by direct statistical analysis of laboratory variation. We previously evaluated our statistical approach for estimating the interlaboratory replicability of a single laboratory discovery. Those results, however, were from a well-coordinated, multi-lab phenotyping study and did not extend to the more realistic setting in which laboratories are operating independently of each other. Here, we sought to test our statistical approach as a realistic prospective experiment, in mice, using 152 results from 5 independent published studies deposited in the Mouse Phenome Database (MPD). In independent replication experiments at 3 laboratories, we found that 53 of the results were replicable, so the other 99 were considered non-replicable. Of the 99 non-replicable results, 59 were statistically significant (at 0.05) in their original single-lab analysis, putting the probability that a single-lab statistical discovery was made even though it is non-replicable, at 59.6%. We then introduced the dimensionless “Genotype-by-Laboratory” (GxL) factor—the ratio between the standard deviations of the GxL interaction and the standard deviation within groups. Using the GxL factor reduced the number of single-lab statistical discoveries and alongside reduced the probability of a non-replicable result to be discovered in the single lab to 12.1%. Such reduction naturally leads to reduced power to make replicable discoveries, but this reduction was small (from 87% to 66%), indicating the small price paid for the large improvement in replicability. Tools and data needed for the above GxL adjustment are publicly available at the MPD and will become increasingly useful as the range of assays and testing conditions in this resource increases.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Department of Statistics and Operations Research, Tel-Aviv University, Tel-Aviv, Israel.
- Department of Epidemiology and Biostatistics, Memorial Sloan Kettering Cancer Center, New York, New York, United States of America.
- School of Zoology, Faculty of Life Sciences, Tel Aviv University, Tel Aviv, Israel.
- The Elton Laboratory for Molecular Neuroendocrinology, Department of Human Molecular Genetics and Biochemistry, Sackler Faculty of Medicine, Sagol School of Neuroscience and Adams Super Center for Brain Studies, Tel Avi…
- The Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, Israel.
- The Jackson Laboratory, Bar Harbor, Maine, United States of America.
ORCID-Profile: Iman Jaljuli, Illana Gozes, Molly A Bogue
Förderung
- NIDA NIH HHS
- United States - Israel Binational Science Foundation
- US–Israel Binational Science Foundation—US National Science Foundation
- US National Institute of Health - US National Science Foundation
Zitieren
APA
Jaljuli, I., Kafkafi, N., Giladi, E., Golani, I., Gozes, I., Chesler, E. J., Bogue, M. A., & Benjamini, Y. (2023). A multi-lab experimental assessment reveals that replicability can be improved by using empirical estimates of genotype-by-lab interaction. PLOS Biology, 21(5), e3002082. https://doi.org/10.1371/journal.pbio.3002082
Vancouver
Jaljuli I, Kafkafi N, Giladi E, Golani I, Gozes I, Chesler EJ, et al. A multi-lab experimental assessment reveals that replicability can be improved by using empirical estimates of genotype-by-lab interaction. PLOS Biology. 2023;21(5):e3002082. doi:10.1371/journal.pbio.3002082
BibTeX
@article{Jaljuli2023,
author = {Jaljuli, Iman and Kafkafi, Neri and Giladi, Eliezer and Golani, Ilan and Gozes, Illana and Chesler, Elissa J and Bogue, Molly A and Benjamini, Yoav},
title = {A multi-lab experimental assessment reveals that replicability can be improved by using empirical estimates of genotype-by-lab interaction},
journal = {PLOS Biology},
year = {2023},
volume = {21},
number = {5},
pages = {e3002082},
doi = {10.1371/journal.pbio.3002082}
}
EUVIMED