Large Language Model–Assisted Risk-of-Bias Assessment in Randomized Controlled Trials Using the Revised Risk-of-Bias Tool: Evaluation Study
Unterstützung von Risiko-of-Bias-Bewertungen in randomisierten kontrollierten Studien mit Large Language Models mit dem revidierten Risk-of-Bias-Tool: Evaluationsstudie
Journal of Medical Internet Research · 2025-06-24 · 27 · e70450
DOI: 10.2196/70450 ↗Auf einen Blick
LLMs erreichten bei RoB2-Bewertungen eine moderate bis hohe Übereinstimmung mit Cochrane- und Reviewer-Bewertungen. Sie bearbeiteten die Bewertungen deutlich schneller als Menschen.
Fragestellung
Wie genau bewerten LLMs das Risiko eines Bias mit RoB2 in RCTs und wie viel Zeit sparen sie gegenüber menschlichen Bewerter:innen?
Methode
Systematische Suche in der Cochrane Library bis Oktober 2023; 46 RCTs aus 86 Reviews; 3 erfahrene Reviewer:innen beurteilten alle RCTs mit RoB2. 6 RCTs dienten der Prompt-Entwicklung.
Ergebnisse
Gegenüber Cochrane Reviews lag die Genauigkeit bei 57,5 % (overall, assignment) und 70 % (overall, adhering). Für die 6 übrigen Domänen lag sie bei 65,2 % (95 %-KI 57,6-72,7) bzw. 74,2 % (95 %-KI 64,7-83,9) gegenüber Reviewer:innen.
Einschränkungen
Nicht alle Domänen erreichten hohe Genauigkeit; die Autoren nennen Bedarf an größeren Studien mit besserem Prompting.
Schlussfolgerung der Autor:innen
LLMs zeigten bei strukturierter Prompteingabe eine beachtliche Genauigkeit, besonders bei strukturierter Verarbeitung methodischer Details. Sie ersetzen die menschliche Bewertung nicht, haben aber Potenzial als Unterstützung.
Kernaussagen
- Genauigkeit je nach Vergleich und RoB2-Domäne unterschiedlich
- Signalfragen: mittlere Genauigkeit 83,2 % (95 %-KI 77,5-88,9)
- Bewertung in 1,9 Minuten vs. 31,5 Minuten bei Menschen
Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.
At a glance
Journal article in Journal of Medical Internet Research, published online on 2025-06-24. Written by Jiajie Huang and 10 co-authors. Institution: Department of Health Policy and Management, School of Public Health, Lanzhou University, Lanzhou, China. Major MEDLINE topics: Randomized Controlled Trials as Topic, Language. Cited 27 times so far (Crossref); 56 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
The revised Risk-of-Bias tool (RoB2) overcomes the limitations of its predecessor but introduces new implementation challenges. Studies demonstrate low interrater reliability and substantial time requirements for RoB2 implementation. Large language models (LLMs) may assist in RoB2 implementation, although their effectiveness remains uncertain.
Objective
This study aims to evaluate the accuracy of LLMs in RoB2 assessments to explore their potential as research assistants for bias evaluation.
Methods
We systematically searched the Cochrane Library (through October 2023) for reviews using RoB2, categorized by interest in adhering or assignment. From 86 eligible reviews of randomized controlled trials (covering 1399 RCTs), we randomly selected 46 RCTs (23 per category). In addition, 3 experienced reviewers independently assessed all 46 RCTs using RoB2, recording assessment time for each trial. Reviewer judgments were reconciled through consensus. Furthermore, 6 RCTs (3 from each category) were randomly selected for prompt development and optimization. The remaining 40 trials established the internal validation standard, while Cochrane Reviews judgments served as external validation. Primary outcomes were extracted as reported in corresponding Cochrane Reviews. We calculated accuracy rates, Cohen κ, and time differentials.
Results
We identified significant differences between Cochrane and reviewer judgments, particularly in domains 1, 4, and 5, likely due to different standards in assessing randomization and blinding. Among the 20 articles focusing on adhering, 18 Cochrane Reviews and 19 reviewer judgments classified them as “High risk,” while assignment-focused RCTs showed more heterogeneous risk distribution. Compared with Cochrane Reviews, LLMs demonstrated accuracy rates of 57.5% and 70% for overall (assignment) and overall (adhering), respectively. When compared with reviewer judgments, LLMs’ accuracy rates were 65% and 70% for these domains. The average accuracy rates for the remaining 6 domains were 65.2% (95% CI 57.6-72.7) against Cochrane Reviews and 74.2% (95% CI 64.7-83.9) against reviewers. At the signaling question level, LLMs achieved 83.2% average accuracy (95% CI 77.5-88.9), with accuracy exceeding 70% for most questions except 2.4 (assignment), 2.5 (assignment), 3.3, and 3.4. When domain judgments were derived from LLM-generated signaling questions using the RoB2 algorithm rather than direct LLM domain judgments, accuracy improved substantially for Domain 2 (adhering; 55-95) and overall (adhering; 70-90). LLMs demonstrated high consistency between iterations (average 85.2%, 95% CI 85.15-88.79) and completed assessments in 1.9 minutes versus 31.5 minutes for human reviewers (mean difference 29.6, 95% CI 25.6-33.6 minutes).
Conclusions
LLMs achieved commendable accuracy when guided by structured prompts, particularly through processing methodological details through structured reasoning. While not replacing human assessment, LLMs demonstrate strong potential for assisting RoB2 evaluations. Larger studies with improved prompting could enhance performance.
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Department of Health Policy and Management, School of Public Health, Lanzhou University, Lanzhou, China.
- Evidence-Based Social Science Research Center, School of Public Health, Lanzhou University, Lanzhou, China.
- School of Nursing, Southern Medical University, Guangzhou, China.
- School of Nursing, Peking University, Beijing, China.
- College of Nursing, Gansu University of Traditional Chinese Medicine, Lanzhou, China.
- Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, China.
ORCID profiles: Jiajie Huang, Honghao Lai, Weilong Zhao, Danni Xia, Chunyang Bai, Mingyao Sun, Jianing Liu, Jiayi Liu, Bei Pan, Jinhui Tian, Long Ge
Cite
APA
Huang, J., Lai, H., Zhao, W., Xia, D., Bai, C., Sun, M., Liu, J., Liu, J., Pan, B., Tian, J., & Ge, L. (2025). Large Language Model–Assisted Risk-of-Bias Assessment in Randomized Controlled Trials Using the Revised Risk-of-Bias Tool: Evaluation Study. Journal of Medical Internet Research, 27, e70450. https://doi.org/10.2196/70450
Vancouver
Huang J, Lai H, Zhao W, Xia D, Bai C, Sun M, et al. Large Language Model–Assisted Risk-of-Bias Assessment in Randomized Controlled Trials Using the Revised Risk-of-Bias Tool: Evaluation Study. Journal of Medical Internet Research. 2025;27:e70450. doi:10.2196/70450
BibTeX
@article{Huang2025,
author = {Huang, Jiajie and Lai, Honghao and Zhao, Weilong and Xia, Danni and Bai, Chunyang and Sun, Mingyao and Liu, Jianing and Liu, Jiayi and Pan, Bei and Tian, Jinhui and Ge, Long},
title = {Large Language Model–Assisted Risk-of-Bias Assessment in Randomized Controlled Trials Using the Revised Risk-of-Bias Tool: Evaluation Study},
journal = {Journal of Medical Internet Research},
year = {2025},
volume = {27},
pages = {e70450},
doi = {10.2196/70450}
}
EUVIMED