What is the probability that higher versus lower quality of evidence represents true effects estimates?
Journal of Evaluation in Clinical Practice · 2025
DOI: 10.1111/jep.14160 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Evaluation in Clinical Practice, 2025. Verfasst von Benjamin Djulbegovic und 5 weiteren Autor:innen. Beteiligte Einrichtung: Division of Hematology/Oncology, Department of Medicine, Medical University of South Carolina, Charleston, South Carolina, USA. Hauptthemen laut MEDLINE: Meta-Analysis as Topic, Evidence-Based Medicine. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 34 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: AHRQ HHS, Agency for Healthcare Research and Quality.
Abstract (Originalsprache)
Rationale, Aims, and Objectives The previous studies demonstrated that the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system, a leading method for evaluating the certainty (quality) of scientific evidence (CoE), cannot reliably differentiate between various levels of CoE when the objective is to accurately assess the magnitude of the treatment effect. An estimated effect size is a function of multiple factors, including the true underlying treatment effect, biases, and other nonlinear factors that affect the estimate in different directions. We postulate that non‐weighted, simple linear tallying can provide more accurate estimates of the probability of a true estimate of treatment effects as a function of CoE.
Methods
We reasoned that stable treatment effect estimates over time indicate truthfulness. We compared odds ratios (ORs) from meta‐analyses (MAs) before and after updates, hypothesising that a ratio of odds ratios (ROR) equal to 1 will be more commonly observed in higher versus lower CoE. We used a subset of a previously analysed data set consisting of 82 Cochrane pairs of MAs in which CoE has not changed with the updated MA. If the linear model is valid, we would expect a decrease in the number of ROR = 1 cases as we move from high to moderate, low, and very low CoE.
Results
We found a linear relationship between the probability of a potentially ‘true’ estimate of treatment effects as a function of CoE (assuming a 10% ROR error margin) ( R 2 = 1; p = 0.001). The probability of potentially ‘true’ estimates decreases by 21% (95% CI: 18%–24%) for each drop in the rating of CoE. A linear relationship with a 5% ROR error margin was less clear, likely due to a smaller sample size. Still, higher CoE showed a significantly greater probability of ‘true’ effects (53%) compared to non‐high (i.e., moderate, low, or very low) CoE (25%); p = 0.032.
Conclusion
This study confirmed linear relationship between CoE and the probability of potentially ‘true’ estimates. We found that the probability of potentially “true” estimates decreases by about 20% for each drop in CoE (from about 80% for high to 55% for moderate to 35% to low and 15% to very low CoE).
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Division of Hematology/Oncology, Department of Medicine, Medical University of South Carolina, Charleston, South Carolina, USA.
- Department of Mathematics, Indiana University Northwest, Gary, Indiana, USA.
- Clinic of Orthodontics and Pediatric Dentistry, Center of Dental Medicine, University of Zurich, Zurich, Switzerland.
- Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, California, USA.
- Nuffield Department of Primary Care Health Sciences, Centre for Evidence-Based Medicine, Oxford, UK.
- Dartmouth Institute for Health Policy and Clinical Practice (TDI) Geisel School of Medicine, Dartmouth College, Hanover, New Hampshire, USA.
ORCID-Profile: Benjamin Djulbegovic
Förderung
- AHRQ HHS
- Agency for Healthcare Research and Quality
Zitieren
APA
Djulbegovic, B., Hozo, I., Koletsi, D., Price, A., Nunan, D., & Hemkens, L. G. (2025). What is the probability that higher versus lower quality of evidence represents true effects estimates?. Journal of Evaluation in Clinical Practice. https://doi.org/10.1111/jep.14160
Vancouver
Djulbegovic B, Hozo I, Koletsi D, Price A, Nunan D, Hemkens LG. What is the probability that higher versus lower quality of evidence represents true effects estimates?. Journal of Evaluation in Clinical Practice. 2025. doi:10.1111/jep.14160
BibTeX
@article{Djulbegovic2025,
author = {Djulbegovic, Benjamin and Hozo, Iztok and Koletsi, Despina and Price, Amy and Nunan, David and Hemkens, Lars G},
title = {What is the probability that higher versus lower quality of evidence represents true effects estimates?},
journal = {Journal of Evaluation in Clinical Practice},
year = {2025},
doi = {10.1111/jep.14160}
}
EUVIMED