EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

What is the probability that higher versus lower quality of evidence represents true effects estimates?

Benjamin Djulbegovic iD, Iztok Hozo, Despina Koletsi, Amy Price, David Nunan, Lars G Hemkens

Journal of Evaluation in Clinical Practice · 2025

DOI: 10.1111/jep.14160 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Evaluation in Clinical Practice, 2025. Verfasst von Benjamin Djulbegovic und 5 weiteren Autor:innen. Beteiligte Einrichtung: Division of Hematology/Oncology, Department of Medicine, Medical University of South Carolina, Charleston, South Carolina, USA. Hauptthemen laut MEDLINE: Meta-Analysis as Topic, Evidence-Based Medicine. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 34 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: AHRQ HHS, Agency for Healthcare Research and Quality.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Rationale, Aims, and Objectives The previous studies demonstrated that the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system, a leading method for evaluating the certainty (quality) of scientific evidence (CoE), cannot reliably differentiate between various levels of CoE when the objective is to accurately assess the magnitude of the treatment effect. An estimated effect size is a function of multiple factors, including the true underlying treatment effect, biases, and other nonlinear factors that affect the estimate in different directions. We postulate that non‐weighted, simple linear tallying can provide more accurate estimates of the probability of a true estimate of treatment effects as a function of CoE.

Methods

We reasoned that stable treatment effect estimates over time indicate truthfulness. We compared odds ratios (ORs) from meta‐analyses (MAs) before and after updates, hypothesising that a ratio of odds ratios (ROR) equal to 1 will be more commonly observed in higher versus lower CoE. We used a subset of a previously analysed data set consisting of 82 Cochrane pairs of MAs in which CoE has not changed with the updated MA. If the linear model is valid, we would expect a decrease in the number of ROR = 1 cases as we move from high to moderate, low, and very low CoE.

Results

We found a linear relationship between the probability of a potentially ‘true’ estimate of treatment effects as a function of CoE (assuming a 10% ROR error margin) ( R 2 = 1; p = 0.001). The probability of potentially ‘true’ estimates decreases by 21% (95% CI: 18%–24%) for each drop in the rating of CoE. A linear relationship with a 5% ROR error margin was less clear, likely due to a smaller sample size. Still, higher CoE showed a significantly greater probability of ‘true’ effects (53%) compared to non‐high (i.e., moderate, low, or very low) CoE (25%); p = 0.032.

Conclusion

This study confirmed linear relationship between CoE and the probability of potentially ‘true’ estimates. We found that the probability of potentially “true” estimates decreases by about 20% for each drop in CoE (from about 80% for high to 55% for moderate to 35% to low and 15% to very low CoE).

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Meta-Analysis as TopicEvidence-Based MedicineHumansProbabilityOdds Ratio

Stichwörter der Autor:innen

evidence‐based medicinehierarchy of evidencetruth

Autor:innen und Institutionen

  • Division of Hematology/Oncology, Department of Medicine, Medical University of South Carolina, Charleston, South Carolina, USA.
  • Department of Mathematics, Indiana University Northwest, Gary, Indiana, USA.
  • Clinic of Orthodontics and Pediatric Dentistry, Center of Dental Medicine, University of Zurich, Zurich, Switzerland.
  • Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Stanford, California, USA.
  • Nuffield Department of Primary Care Health Sciences, Centre for Evidence-Based Medicine, Oxford, UK.
  • Dartmouth Institute for Health Policy and Clinical Practice (TDI) Geisel School of Medicine, Dartmouth College, Hanover, New Hampshire, USA.

ORCID-Profile: Benjamin Djulbegovic

Förderung

  • AHRQ HHS
  • Agency for Healthcare Research and Quality

Zitieren

APA

Djulbegovic, B., Hozo, I., Koletsi, D., Price, A., Nunan, D., & Hemkens, L. G. (2025). What is the probability that higher versus lower quality of evidence represents true effects estimates?. Journal of Evaluation in Clinical Practice. https://doi.org/10.1111/jep.14160

Vancouver

Djulbegovic B, Hozo I, Koletsi D, Price A, Nunan D, Hemkens LG. What is the probability that higher versus lower quality of evidence represents true effects estimates?. Journal of Evaluation in Clinical Practice. 2025. doi:10.1111/jep.14160

BibTeX
@article{Djulbegovic2025,
  author = {Djulbegovic, Benjamin and Hozo, Iztok and Koletsi, Despina and Price, Amy and Nunan, David and Hemkens, Lars G},
  title = {What is the probability that higher versus lower quality of evidence represents true effects estimates?},
  journal = {Journal of Evaluation in Clinical Practice},
  year = {2025},
  doi = {10.1111/jep.14160}
}
Kostenlos starten →