Exploring the potential of Claude 2 for risk of bias assessment: Using a large language model to assess randomized controlled trials with RoB 2
Research Synthesis Methods · 2025
DOI: 10.1017/rsm.2025.12 ↗Kurz gefasst
Fachartikel in der Zeitschrift Research Synthesis Methods, 2025. Verfasst von Angelika Eisele-Metzger und 7 weiteren Autor:innen. Beteiligte Einrichtung: Institute for Evidence in Medicine, Medical Center, Faculty of Medicine, University of Freiburg, Freiburg, Germany. Hauptthemen laut MEDLINE: Large Language Models, Randomized Controlled Trials as Topic. Die Arbeit wurde bisher 19-mal zitiert (Crossref) und nennt 73 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Systematic reviews are essential for evidence-based health care, but conducting them is time- and resource-consuming. To date, efforts have been made to accelerate and (semi-)automate various steps of systematic reviews through the use of artificial intelligence (AI) and the emergence of large language models (LLMs) promises further opportunities. One crucial but complex task within systematic review conduct is assessing the risk of bias (RoB) of included studies. Therefore, the aim of this study was to test the LLM Claude 2 for RoB assessment of 100 randomized controlled trials, published in English language from 2013 onwards, using the revised Cochrane risk of bias tool (‘RoB 2’; involving judgements for five specific domains and an overall judgement). We assessed the agreement of RoB judgements by Claude with human judgements published in Cochrane reviews. The observed agreement between Claude and Cochrane authors ranged from 41% for the overall judgement to 71% for domain 4 (‘outcome measurement’). Cohen’s κ was lowest for domain 5 (‘selective reporting’; 0.10 (95% confidence interval (CI): −0.10–0.31)) and highest for domain 3 (‘missing data’; 0.31 (95% CI: 0.10–0.52)), indicating slight to fair agreement. Fair agreement was found for the overall judgement (Cohen’s κ: 0.22 (95% CI: 0.06–0.38)). Sensitivity analyses using alternative prompting techniques or the more recent version Claude 3 did not result in substantial changes. Currently, Claude’s RoB 2 judgements cannot replace human RoB assessment. However, the potential of LLMs to support RoB assessment should be further explored.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Institute for Evidence in Medicine, Medical Center, Faculty of Medicine, University of Freiburg, Freiburg, Germany.
- Cochrane Germany, Cochrane Germany Foundation, Freiburg, Germany.
- Eye Center, Medical Center, Faculty of Medicine, University of Freiburg, Freiburg, Germany.
- Institute for Digitalization in Medicine, Medical Center, Faculty of Medicine, University of Freiburg, Freiburg, Germany.
Zitieren
APA
Eisele-Metzger, A., Lieberum, J. L., Toews, M., Siemens, W., Heilmeyer, F., Haverkamp, C., Boehringer, D., & Meerpohl, J. J. (2025). Exploring the potential of Claude 2 for risk of bias assessment: Using a large language model to assess randomized controlled trials with RoB 2. Research Synthesis Methods. https://doi.org/10.1017/rsm.2025.12
Vancouver
Eisele-Metzger A, Lieberum JL, Toews M, Siemens W, Heilmeyer F, Haverkamp C, et al. Exploring the potential of Claude 2 for risk of bias assessment: Using a large language model to assess randomized controlled trials with RoB 2. Research Synthesis Methods. 2025. doi:10.1017/rsm.2025.12
BibTeX
@article{EiseleMetzger2025,
author = {Eisele-Metzger, Angelika and Lieberum, Judith-Lisa and Toews, Markus and Siemens, Waldemar and Heilmeyer, Felix and Haverkamp, Christian and Boehringer, Daniel and Meerpohl, Joerg J},
title = {Exploring the potential of Claude 2 for risk of bias assessment: Using a large language model to assess randomized controlled trials with RoB 2},
journal = {Research Synthesis Methods},
year = {2025},
doi = {10.1017/rsm.2025.12}
}
EUVIMED