EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Don't Stop Me Now, `Cause I'm Having a Good Time Screening: Evaluation of Stopping Methods for Safe Use of Priority Screening in Systematic Reviews

Tim Repke iD, Francesca Tinsdeall, Diana Danilenko, Sergio Graziosi iD, Finn Müller-Hansen, Lena Schmidt, James Thomas iD, Gert van Valkenhoef

Cochrane Evidence Synthesis and Methods · 2026-01 · 4 (1)

DOI: 10.1002/cesm.70068 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Cochrane Evidence Synthesis and Methods, online erschienen am 21.01.2026. Verfasst von Tim Repke und 7 weiteren Autor:innen. Beteiligte Einrichtung: Climate Economics and Policy Potsdam Institute for Climate Impact Research (PIK), Member of the Leibniz Association Potsdam Germany. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 64 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Wellcome Trust, European Research Council.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Introduction

Priority screening has the potential to reduce the number of records that need to be annotated in systematic literature reviews. So‐called technology‐assisted reviews (TAR) use machine‐learning with prior include/exclude annotations to continuously rank unseen records by their predicted relevance to find relevant records earlier. In this article, we present a systematic evaluation of methods to determine when it is safe to stop screening when using prioritization.

Methods

We implement an open‐source evaluation framework that features a novel method to generate rankings and simulate priority screening processes for 81 real‐world data sets. We use these simulations to evaluate 15 statistical or rule‐based (heuristic) stopping methods, testing a range of hyperparameters for each.

Results

The work‐saving potential and performance of stopping criteria heavily rely on “good” rankings, which are typically not achieved by a single ranking algorithm across the entire screening process. Our evaluation shows that almost all existing stopping methods either fail to reliably stop without missing relevant records or fail to utilize the full potential work‐savings. Only one method reliably meets the set recall target, but stops conservatively.

Conclusions

Many digital evidence synthesis tools provide priority screening features that are already used in many research projects. However, the theoretical work‐savings demonstrated in retrospective simulations of prioritization can only be unlocked with safe and reproducible stopping criteria. Our results highlight the need for improved stopping methods and guidelines on how to responsibly use priority screening. We also urge screening platforms to provide indicators and authors to transparently report metrics when automating (parts of) their synthesis.

Schlagwörter

Stichwörter der Autor:innen

digital evidence synthesispriority screeningstopping methodssystematic mapssystematic reviewstechnology‐assisted reviews

Autor:innen und Institutionen

  • Climate Economics and Policy Potsdam Institute for Climate Impact Research (PIK), Member of the Leibniz Association Potsdam Germany.
  • Evidence for Policy and Practice Information Center, UCL Social Research Institute, Institute of Education University College London London UK.
  • Centre for Clinical Brain Sciences University of Edinburgh Edinburgh UK.
  • NIHR Innovation Observatory Newcastle University Newcastle upon Tyne UK.
  • Central Executive Team Cochrane Collaboration London UK.

ORCID-Profile: Tim Repke, Sergio Graziosi, James Thomas

Förderung

  • Wellcome Trust
  • European Research Council

Zitieren

APA

Repke, T., Tinsdeall, F., Danilenko, D., Graziosi, S., Müller-Hansen, F., Schmidt, L., Thomas, J., & van Valkenhoef, G. (2026). Don't Stop Me Now, `Cause I'm Having a Good Time Screening: Evaluation of Stopping Methods for Safe Use of Priority Screening in Systematic Reviews. Cochrane Evidence Synthesis and Methods, 4(1). https://doi.org/10.1002/cesm.70068

Vancouver

Repke T, Tinsdeall F, Danilenko D, Graziosi S, Müller-Hansen F, Schmidt L, et al. Don't Stop Me Now, `Cause I'm Having a Good Time Screening: Evaluation of Stopping Methods for Safe Use of Priority Screening in Systematic Reviews. Cochrane Evidence Synthesis and Methods. 2026;4(1). doi:10.1002/cesm.70068

BibTeX
@article{Repke2026,
  author = {Repke, Tim and Tinsdeall, Francesca and Danilenko, Diana and Graziosi, Sergio and Müller-Hansen, Finn and Schmidt, Lena and Thomas, James and van Valkenhoef, Gert},
  title = {Don't Stop Me Now, `Cause I'm Having a Good Time Screening: Evaluation of Stopping Methods for Safe Use of Priority Screening in Systematic Reviews},
  journal = {Cochrane Evidence Synthesis and Methods},
  year = {2026},
  volume = {4},
  number = {1},
  doi = {10.1002/cesm.70068}
}
Kostenlos starten →