EUVIMEDEUVIMED entdecken
Vergleichsstudie

Towards the automatic risk of bias assessment on randomized controlled trials: A comparison of RobotReviewer and humans

Yuan Tian, Xi Yang, Suhail A Doi, Luis Furuya-Kanamori iD, Lifeng Lin iD, Joey S W Kwong, Chang Xu iD

Research Synthesis Methods · 2024

DOI: 10.1002/jrsm.1761 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift Research Synthesis Methods, 2024. Verfasst von Yuan Tian und 6 weiteren Autor:innen. Beteiligte Einrichtung: First School of Clinical Medicine, Anhui Medical University, Hefei, China. Hauptthemen laut MEDLINE: Randomized Controlled Trials as Topic, Bias. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 22 Literaturangaben. Gefördert von: Shanghai Eastern Hepatobiliary Surgery Hospital of Naval Medical University, National Natural Science Foundation of China, Qatar National Research Fund.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

RobotReviewer is a tool for automatically assessing the risk of bias in randomized controlled trials, but there is limited evidence of its reliability. We evaluated the agreement between RobotReviewer and humans regarding the risk of bias assessment based on 1955 randomized controlled trials. The risk of bias in these trials was assessed via two different approaches: (1) manually by human reviewers, and (2) automatically by the RobotReviewer. The manual assessment was based on two groups independently, with two additional rounds of verification. The agreement between RobotReviewer and humans was measured via the concordance rate and Cohen's kappa statistics, based on the comparison of binary classification of the risk of bias (low vs. high/unclear) as restricted by RobotReviewer. The concordance rates varied by domain, ranging from 63.07% to 83.32%. Cohen's kappa statistics showed a poor agreement between humans and RobotReviewer for allocation concealment ( κ = 0.25, 95% CI: 0.21–0.30), blinding of outcome assessors ( κ = 0.27, 95% CI: 0.23–0.31); While moderate for random sequence generation ( κ = 0.46, 95% CI: 0.41–0.50) and blinding of participants and personnel ( κ = 0.59, 95% CI: 0.55–0.64). The findings demonstrate that there were domain‐specific differences in the level of agreement between RobotReviewer and humans. We suggest that it might be a useful auxiliary tool, but the specific manner of its integration as a complementary tool requires further discussion.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Randomized Controlled Trials as TopicBiasHumansReproducibility of ResultsSoftwareAlgorithmsResearch DesignRisk AssessmentAutomation

Stichwörter der Autor:innen

RobotReviewerautomationrapid reviewrisk of bias

Autor:innen und Institutionen

  • First School of Clinical Medicine, Anhui Medical University, Hefei, China.
  • Department of Maternal, Child and Adolescent Health, School of Public Health, Anhui Medical University, Hefei, China.
  • Department of Population Medicine, College of Medicine, QU Health, Qatar University, Doha, Qatar.
  • UQ Centre for Clinical Research, The University of Queensland, Herston, Queensland, Australia.
  • Department of Epidemiology and Biostatistics, University of Arizona, Tucson, Arizona, USA.
  • Global Health Nursing, Graduate School of Nursing Science, St Luke's International University, Tokyo, Japan.

ORCID-Profile: Luis Furuya-Kanamori, Lifeng Lin, Chang Xu

Förderung

  • Shanghai Eastern Hepatobiliary Surgery Hospital of Naval Medical University
  • National Natural Science Foundation of China
  • Qatar National Research Fund

Zitieren

APA

Tian, Y., Yang, X., Doi, S. A., Furuya-Kanamori, L., Lin, L., Kwong, J. S. W., & Xu, C. (2024). Towards the automatic risk of bias assessment on randomized controlled trials: A comparison of RobotReviewer and humans. Research Synthesis Methods. https://doi.org/10.1002/jrsm.1761

Vancouver

Tian Y, Yang X, Doi SA, Furuya-Kanamori L, Lin L, Kwong JSW, et al. Towards the automatic risk of bias assessment on randomized controlled trials: A comparison of RobotReviewer and humans. Research Synthesis Methods. 2024. doi:10.1002/jrsm.1761

BibTeX
@article{Tian2024,
  author = {Tian, Yuan and Yang, Xi and Doi, Suhail A and Furuya-Kanamori, Luis and Lin, Lifeng and Kwong, Joey S W and Xu, Chang},
  title = {Towards the automatic risk of bias assessment on randomized controlled trials: A comparison of RobotReviewer and humans},
  journal = {Research Synthesis Methods},
  year = {2024},
  doi = {10.1002/jrsm.1761}
}
Kostenlos starten →