Evaluating the Accuracy of Large Language Models in Risk-of-Bias Assessment Using Version 2 of the Cochrane Risk-of-Bias Tool for Randomized Trials: Exploratory Feasibility Study
Journal of Medical Internet Research · 25.09.2026 · 28 · e84915-e84915
DOI: 10.2196/84915 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Medical Internet Research, online erschienen am 25.09.2026. Verfasst von Yu-Ju Lai und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Pharmacy, Fu Jen Catholic University Hospital, Fu Jen Catholic University, No.69, Guizi Rd., Taishan Dist., New Taipei City, 24352, Tai…. Hauptthemen laut MEDLINE: Randomized Controlled Trials as Topic. Die Arbeit nennt 60 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Large language models (LLMs) have the potential to improve the efficiency of evidence synthesis, but their reliability in performing complex tasks such as risk-of-bias (ROB) assessment in randomized controlled trials (RCTs) remains unclear.
Objective
This study aimed to evaluate whether LLMs can reliably assess ROB in RCTs using version 2 of the Cochrane ROB tool for randomized trials (ROB 2).
Methods
This study was conducted between December 28, 2024, and February 28, 2025, in adherence to American Association for Public Opinion Research reporting guidelines. Twenty-nine RCTs were selected from published Cochrane systematic reviews across diverse medical fields. We developed a structured prompt engineering framework that transformed ROB 2 decision trees into logical rules for the LLM. Each RCT was independently evaluated twice by ChatGPT, with Cochrane review authors’ assessments serving as the reference standard for comparison. The main outcomes were the accuracy and consistency of ROB 2 assessments at both the domain and trial levels, evaluated using accuracy, sensitivity, specificity, and F 1 -score. Consistency between the repeated assessments was quantified using the Cohen κ and prevalence-adjusted, bias-adjusted κ.
Results
The LLM demonstrated a moderate aggregate domain accuracy of 73.1% (95% CI 64.7%-81.5%) in the first assessment and 75.9% (95% CI 66.3%-85.4%) in the second assessment. Domain-averaged sensitivity decreased from 61.4% (95% CI 48.1%‐74.7%) to 53.4% (95% CI 41.7%‐65.0%), whereas domain-averaged specificity increased from 75.8% (95% CI 65.3%-86.3%) to 81.1% (95%CI 67.1%-95%), indicating a conservative tendency in identifying a high ROB. Domain-level accuracy ranged from 62.1% to 87.9%, with the lowest accuracy observed in domain 1 and the lowest F 1 -scores observed in domain 2. Consistency between repeated assessments was high, with a mean agreement of 89.0% (SD 7.5%), and Cohen κ values were 0.86, 0.39, 0.56, 0.84, and 0.85 in domains 1 to 5, respectively.
Conclusions
In this exploratory study, ChatGPT demonstrated moderate accuracy and high consistency in assessing ROB in RCTs using the ROB 2. However, its reliability diminished in complex scenarios requiring interpretation of implicit narratives or behavioral nuance. These findings suggest that LLMs may support methodological evaluations in systematic reviews by acting as automated screeners to reduce reviewer burden, but current implementation still requires expert oversight, particularly for trials involving subjective outcomes or nonstandard reporting.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Pharmacy, Fu Jen Catholic University Hospital, Fu Jen Catholic University, No.69, Guizi Rd., Taishan Dist., New Taipei City, 24352, Taiwan.
- Graduate Institute of Clinical Medicine, College of Medicine, Taipei Medical University, Taipei, Taiwan.
Zitieren
APA
Lai, Y. J., Lin, S. H., & Liu, J. W. (2026). Evaluating the Accuracy of Large Language Models in Risk-of-Bias Assessment Using Version 2 of the Cochrane Risk-of-Bias Tool for Randomized Trials: Exploratory Feasibility Study. Journal of Medical Internet Research, 28, e84915-e84915. https://doi.org/10.2196/84915
Vancouver
Lai YJ, Lin SH, Liu JW. Evaluating the Accuracy of Large Language Models in Risk-of-Bias Assessment Using Version 2 of the Cochrane Risk-of-Bias Tool for Randomized Trials: Exploratory Feasibility Study. Journal of Medical Internet Research. 2026;28:e84915-e84915. doi:10.2196/84915
BibTeX
@article{Lai2026,
author = {Lai, Yu-Ju and Lin, Shen-Hua and Liu, Jen-Wei},
title = {Evaluating the Accuracy of Large Language Models in Risk-of-Bias Assessment Using Version 2 of the Cochrane Risk-of-Bias Tool for Randomized Trials: Exploratory Feasibility Study},
journal = {Journal of Medical Internet Research},
year = {2026},
volume = {28},
pages = {e84915--e84915},
doi = {10.2196/84915}
}
EUVIMED