Exploring the Garden of Forking Paths in Empirical Software Engineering Research: A Multiverse Analysis
ACM Transactions on Software Engineering and Methodology · 31.10.2026 · 35 (10) · 1-26
DOI: 10.1145/3799711 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Software Engineering and Methodology, online erschienen am 16.09.2026. Verfasst von Nathan Cassee und 1 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 73 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Science Council, “Automated Boundary Testing for AI/ML models” and from WASP for the project “Bound-Miner.”.
Abstract (Originalsprache)
In empirical software engineering (SE) research, researchers have considerable freedom to decide how to process data, what operationalizations to use, and which statistical model to fit. Gelman and Loken refer to this freedom as leading to a “garden of forking paths.” Although this freedom is often seen as an advantage, it also poses a threat to robustness and replicability: variations in analytical decisions, even when justifiable, can lead to divergent conclusions. To better understand this risk, we conducted a so-called multiverse analysis on a published empirical SE paper. The paper we picked is a Mining Software Repositories study, as MSR studies commonly use non-trivial statistical models to analyze post hoc, observational data. In the study, we identified nine pivotal analytical decisions—each with at least one equally defensible alternative—and systematically reran all the 3,072 resulting analysis pipelines on the original dataset. Interestingly, only six of these universes (
Förderung
- Science Council
- “Automated Boundary Testing for AI/ML models” and from WASP for the project “Bound-Miner.”
Zitieren
APA
Cassee, N., & Feldt, R. (2026). Exploring the Garden of Forking Paths in Empirical Software Engineering Research: A Multiverse Analysis. ACM Transactions on Software Engineering and Methodology, 35(10), 1-26. https://doi.org/10.1145/3799711
Vancouver
Cassee N, Feldt R. Exploring the Garden of Forking Paths in Empirical Software Engineering Research: A Multiverse Analysis. ACM Transactions on Software Engineering and Methodology. 2026;35(10):1-26. doi:10.1145/3799711
BibTeX
@article{Cassee2026,
author = {Cassee, Nathan and Feldt, Robert},
title = {Exploring the Garden of Forking Paths in Empirical Software Engineering Research: A Multiverse Analysis},
journal = {ACM Transactions on Software Engineering and Methodology},
year = {2026},
volume = {35},
number = {10},
pages = {1--26},
doi = {10.1145/3799711}
}
EUVIMED