EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Computational reproducibility of Jupyter notebooks from biomedical publications

Sheeba Samuel iD, Daniel Mietchen iD

GigaScience · 02.01.2024 · 13

DOI: 10.1093/gigascience/giad113 ↗

Kurz gefasst

Fachartikel in der Zeitschrift GigaScience, online erschienen am 11.01.2024. Verfasst von Sheeba Samuel und 1 weiteren Autor:innen. Beteiligte Einrichtung: Heinz-Nixdorf Chair for Distributed Information Systems, Friedrich Schiller University Jena, Jena 07743, Germany. Hauptthemen laut MEDLINE: Documentation, Records. Die Arbeit wurde bisher 54-mal zitiert (Crossref) und nennt 158 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Alfred P. Sloan Foundation, Deutsche Forschungsgemeinschaft.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Jupyter notebooks facilitate the bundling of executable code with its documentation and output in one interactive environment, and they represent a popular mechanism to document and share computational workflows, including for research publications. The reproducibility of computational aspects of research is a key component of scientific reproducibility but has not yet been assessed at scale for Jupyter notebooks associated with biomedical publications. Approach We address computational reproducibility at 2 levels: (i) using fully automated workflows, we analyzed the computational reproducibility of Jupyter notebooks associated with publications indexed in the biomedical literature repository PubMed Central. We identified such notebooks by mining the article’s full text, trying to locate them on GitHub, and attempting to rerun them in an environment as close to the original as possible. We documented reproduction success and exceptions and explored relationships between notebook reproducibility and variables related to the notebooks or publications. (ii) This study represents a reproducibility attempt in and of itself, using essentially the same methodology twice on PubMed Central over the course of 2 years, during which the corpus of Jupyter notebooks from articles indexed in PubMed Central has grown in a highly dynamic fashion.

Results

Out of 27,271 Jupyter notebooks from 2,660 GitHub repositories associated with 3,467 publications, 22,578 notebooks were written in Python, including 15,817 that had their dependencies declared in standard requirement files and that we attempted to rerun automatically. For 10,388 of these, all declared dependencies could be installed successfully, and we reran them to assess reproducibility. Of these, 1,203 notebooks ran through without any errors, including 879 that produced results identical to those reported in the original notebook and 324 for which our results differed from the originally reported ones. Running the other notebooks resulted in exceptions.

Conclusions

We zoom in on common problems and practices, highlight trends, and discuss potential improvements to Jupyter-related workflows associated with biomedical publications.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

DocumentationRecordsReproducibility of ResultsReproductionWorkflow

Stichwörter der Autor:innen

GitHubJupyter notebooksPubMed CentralPythoncomputational reproducibilitydependency decayworkflow documentation

Autor:innen und Institutionen

  • Heinz-Nixdorf Chair for Distributed Information Systems, Friedrich Schiller University Jena, Jena 07743, Germany.
  • Michael Stifel Center Jena, Jena 07743, Germany.
  • Ronin Institute, Montclair 07043-2314, NJ, United States.
  • Institute for Globally Distributed Open Research and Education (IGDORE).
  • FIZ Karlsruhe-Leibniz Institute for Information Infrastructure, Berlin 76344, Germany.

ORCID-Profile: Sheeba Samuel, Daniel Mietchen

Förderung

  • Alfred P. Sloan Foundation
  • Deutsche Forschungsgemeinschaft

Zitieren

APA

Samuel, S., & Mietchen, D. (2024). Computational reproducibility of Jupyter notebooks from biomedical publications. GigaScience, 13. https://doi.org/10.1093/gigascience/giad113

Vancouver

Samuel S, Mietchen D. Computational reproducibility of Jupyter notebooks from biomedical publications. GigaScience. 2024;13. doi:10.1093/gigascience/giad113

BibTeX
@article{Samuel2024,
  author = {Samuel, Sheeba and Mietchen, Daniel},
  title = {Computational reproducibility of Jupyter notebooks from biomedical publications},
  journal = {GigaScience},
  year = {2024},
  volume = {13},
  doi = {10.1093/gigascience/giad113}
}
Kostenlos starten →