Assessing the Reliability of Large Language Models for Evaluation of Risk-of-Bias in Randomized Clinical Trials
American Journal of Perinatology · 2026-08 · 43 (11) · 1527-1532
DOI: 10.1055/a-2793-9092 ↗Kurz gefasst
Fachartikel in der Zeitschrift American Journal of Perinatology, online erschienen am 05.03.2026. Verfasst von Takeshi Nagao und 1 weiteren Autor:innen. Beteiligte Einrichtung: Department of Obstetrics and Gynecology, The Jikei University School of Medicine, Tokyo, Japan. Hauptthemen laut MEDLINE: Large Language Models, Randomized Controlled Trials as Topic.
Abstract (Originalsprache)
Objective
Systematic reviews depend on rigorous risk-of-bias (RoB) assessments to ensure credibility, yet manual evaluation using the Cochrane RoB 2 tool is resource-intensive. While Large Language Models (LLMs) offer potential for automation, their alignment with human judgment remains underexplored. This study evaluates the reliability of ChatGPT-4o, ChatGPT-5, and Claude 3.5 Sonnet in assessing RoB in randomized controlled trials (RCTs), comparing their agreement with human reviewers and internal consistency. Study Design We retrospectively analyzed 180 RCTs from systematic reviews published in the American Journal of Obstetrics and Gynecology (2021–2023) reporting complete human RoB 2 ratings. Each LLM processed full-text PDFs using a standardized prompt incorporating the complete RoB 2 algorithm. Model performance was evaluated against human benchmarks using Cohen’s kappa and prevalence- and bias-adjusted kappa (PABAK). Intra-model reliability was assessed across three independent runs to measure consistency.
Results
ChatGPT-5 consistently outperformed other models, achieving the highest agreement in randomization (Domain 1; 76%), missing outcome data (Domain 3; 80%), and outcome measurement (Domain 4; 76%). It showed moderate concordance for deviations from intended interventions (69%). However, all models struggled with selective reporting (Domain 5), where agreement dropped to 47–51%. For overall risk-of-bias judgments, ChatGPT-5 demonstrated superior concordance (60–62%, κ=0.36–0.40) compared to ChatGPT-4o (45%) and Claude 3.5 Sonnet (43%). ChatGPT-5 also exhibited substantial to near-perfect internal consistency.
Conclusion
Among the evaluated models, ChatGPT-5 most closely approximated human RoB 2 assessments and achieved superior internal consistency, suggesting it could serve as a practical first-pass tool to reduce reviewer burden. However, persistent limitations in detecting selective reporting—likely due to the inability to cross-reference external trial registries—highlight that expert human oversight remains essential for accurate evidence synthesis.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Department of Obstetrics and Gynecology, The Jikei University School of Medicine, Tokyo, Japan.
- Department of Obstetrics and Gynecology, Macon and Joan Brock Virginia Health Sciences Eastern Virginia Medical School at Old Dominion University, Norfolk, Virginia, United States.
ORCID-Profile: Tetsuya Kawakita
Zitieren
APA
Nagao, T., & Kawakita, T. (2026). Assessing the Reliability of Large Language Models for Evaluation of Risk-of-Bias in Randomized Clinical Trials. American Journal of Perinatology, 43(11), 1527-1532. https://doi.org/10.1055/a-2793-9092
Vancouver
Nagao T, Kawakita T. Assessing the Reliability of Large Language Models for Evaluation of Risk-of-Bias in Randomized Clinical Trials. American Journal of Perinatology. 2026;43(11):1527-1532. doi:10.1055/a-2793-9092
BibTeX
@article{Nagao2026,
author = {Nagao, Takeshi and Kawakita, Tetsuya},
title = {Assessing the Reliability of Large Language Models for Evaluation of Risk-of-Bias in Randomized Clinical Trials},
journal = {American Journal of Perinatology},
year = {2026},
volume = {43},
number = {11},
pages = {1527--1532},
doi = {10.1055/a-2793-9092}
}
EUVIMED