Consensus operational criteria for including systematic reviews in umbrella reviews: an international Delphi study
JBI Evidence Synthesis · 28.07.2026
DOI: 10.11124/jbies-26-00181 ↗Kurz gefasst
Delphi-Studie in der Zeitschrift JBI Evidence Synthesis, online erschienen am 28.07.2026. Verfasst von Beth Nichol und 8 weiteren Autor:innen. Beteiligte Einrichtung: School of Communities and Education, Northumbria University, Newcastle upon Tyne, UK and Policy Research Unit Behavioural and Social Sciences, Newcas…. Die Arbeit nennt 36 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).
Abstract (Originalsprache)
Objective
This international, multidisciplinary study aimed to develop consensus-based operational criteria for umbrella reviewers to apply when including systematic reviews.
Introduction
As the volume of academic publications continues to increase, umbrella reviews (systematic reviews of systematic reviews) are increasingly used to synthesize large bodies of evidence and provide high-level overviews. However, in the absence of formal guidance, the inclusion of articles based on study type within umbrella reviews is inconsistent, leading to heterogenous criteria for the inclusion of systematic reviews.
Methods
A classic Delphi method was conducted anonymously online with an expert panel in systematic and umbrella reviews. Experts had published at least 5 umbrella or systematic reviews as first author or at least 10 at any authorship position. In round 1, 68 experts rated agreement with 188 elements of a systematic review identified in a previous study. Elements achieving ≥80% agreement progressed to round 2, where experts rated their certainty regarding inclusion in the operational criteria. Statements were collated into draft criteria and refined across rounds 3 to 5. Free-text comments were incorporated throughout. The consensus threshold was ≥80% initially and reduced to ≥70% after round 4.
Results
Thirty-six experts completed all 5 rounds. In round 1, 25 elements met the ≥80% agreement threshold and were condensed for round 2. In round 2, 19 statements reached ≥80% certainty for inclusion and were combined into operational criteria in round 3, with consensus achieved in round 4 (70.3%). Agreement decreased in round 5 (66.7%), indicating a ceiling effect.
Conclusions
The final consensus comprises criteria distinguishing systematic reviews as explicit and reproducible. Seven criteria are divisible into 12 individual checklist items, spanning the conduct and reporting of an article’s rationale; overall methods; inclusion criteria; search strategy; and approach to screening, extraction, and synthesis. While some characteristics were contested (including operationalizing independent screening and a comprehensive search), the resulting operational criteria provide a practical tool to support transparent and consistent inclusion decisions in umbrella reviews. Review registration: OSF https://osf.io/pxtbd
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- School of Communities and Education, Northumbria University, Newcastle upon Tyne, UK and Policy Research Unit Behavioural and Social Sciences, Newcastle University, Newcastle upon Tyne, UK.
- Psychology,Northumbria University, Newcastle upon Tyne, UK.
- School of Healthcare and Nursing Sciences, Northumbria University, Newcastle upon Tyne, UK.
- Health Evidence Synthesis, Recommendations and Impact (HESRI), School of Public Health, Adelaide University, Adelaide, SA, Australia.
- Science for All, Australia.
- School of Communities and Education, Northumbria University, Newcastle upon Tyne, UK.
ORCID-Profile: Beth Nichol, Jemma McCready, Rebecca Gibson, Goran Erfani, Danielle Pollock, Zachary Munn, Jack S Nunn, Marco Tomietto, Catherine Haighton
Zitieren
APA
Nichol, B., McCready, J., Gibson, R., Erfani, G., Pollock, D., Munn, Z., Nunn, J. S., Tomietto, M., & Haighton, C. (2026). Consensus operational criteria for including systematic reviews in umbrella reviews: an international Delphi study. JBI Evidence Synthesis. https://doi.org/10.11124/jbies-26-00181
Vancouver
Nichol B, McCready J, Gibson R, Erfani G, Pollock D, Munn Z, et al. Consensus operational criteria for including systematic reviews in umbrella reviews: an international Delphi study. JBI Evidence Synthesis. 2026. doi:10.11124/jbies-26-00181
BibTeX
@article{Nichol2026,
author = {Nichol, Beth and McCready, Jemma and Gibson, Rebecca and Erfani, Goran and Pollock, Danielle and Munn, Zachary and Nunn, Jack S and Tomietto, Marco and Haighton, Catherine},
title = {Consensus operational criteria for including systematic reviews in umbrella reviews: an international Delphi study},
journal = {JBI Evidence Synthesis},
year = {2026},
doi = {10.11124/jbies-26-00181}
}
EUVIMED