A new peer reviewer? Comparing AI with human performance in randomized controlled trial risk-of-bias assessment
Advances in Clinical and Experimental Medicine · 2026
DOI: 10.17219/acem/216070 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Advances in Clinical and Experimental Medicine, 2026. Verfasst von Jonathan Lettner und 5 weiteren Autor:innen. Beteiligte Einrichtung: Faculty of Health Sciences Brandenburg, Brandenburg Medical School Theodor Fontane, Brandenburg an der Havel, Germany. Hauptthemen laut MEDLINE: Artificial Intelligence, Randomized Controlled Trials as Topic. Die Arbeit wurde bisher 1-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY 3.0).
Abstract (Originalsprache)
BACKGROUND: Risk-of-bias (RoB) assessment is essential for evidence synthesis but remains time-consuming and inherently subjective. Artificial intelligence (AI) may improve the efficiency of systematic reviews; however, its reliability in reproducing expert RoB judgements remains uncertain. OBJECTIVES: To compare the performance of AI models and human raters in RoB assessment of randomized controlled trials (RCTs) using the revised Joanna Briggs Institute (JBI) critical appraisal tool. MATERIAL AND METHODS: Thirteen RCTs published between 2023 and 2025 in orthopedic journals were independently assessed by 2 human raters (an expert (R1) and a novice (R2)) and 2 AI models (ChatGPT-4.0 (CGPT) and DeepSeek-R1 (DS)) using the 13-domain JBI checklist. Deep-reasoning functionalities (e.g., chain-of-thought prompting) were applied. Inter-rater agreement, deviations from the expert assessment (reference standard), and binary disagreements (e.g., Yes vs No) were analyzed to evaluate consistency. RESULTS: The AI models demonstrated high inter-model agreement (91%), exceeding human-AI agreement (CGPT vs R1: 64%; DS vs R1: 68%). However, both AI systems showed substantial divergence from expert judgements in interpretive domains, including allocation concealment (Q2), blinding (Q7), and overall trial design (Q13), with deviation rates ranging from 30% to 38.5%. Binary decision reversals were more frequent in AI assessments (CGPT: 8.9%; DS: 7.7%) than in the human comparison (R2 vs R1: 2.4%). Human raters showed stronger agreement in contextual interpretation (R1-R2: 89.3%), whereas AI models performed better in rule-based domains (Q8/Q9: 100% agreement). CONCLUSIONS: AI can reliably support the automation of objective components of RoB assessment but remains limited in handling interpretive, context-dependent judgements. A hybrid approach combining AI-assisted pre-screening with expert evaluation may enhance the scalability of systematic reviews without compromising methodological rigor.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Faculty of Health Sciences Brandenburg, Brandenburg Medical School Theodor Fontane, Brandenburg an der Havel, Germany.
- Center of Orthopaedics and Traumatology, University Hospital Brandenburg/Havel, Brandenburg Medical School Theodor Fontane, Brandenburg an der Havel, Germany.
- Osteon Orthopedics and Sports Medicine Clinic, Mostar, Bosnia and Herzegovina.
- Physiotherapy Research Laboratory, University Centre of Physiotherapy and Rehabilitation, Faculty of Physiotherapy, Wroclaw Medical University, Poland.
- JBI Centre for Evidence-Based Healthcare Poland, University of Adelaide, Australia.
- Department of Orthopedics and Traumatology, University of Health Sciences, Istanbul Fatih Sultan Mehmet Training and Research Hospital, Turkey.
Zitieren
APA
Lettner, J., Ostojic, M., Królikowska, A., Kayaalp, M. E., Ramadanov, N., & Prill, R. (2026). A new peer reviewer? Comparing AI with human performance in randomized controlled trial risk-of-bias assessment. Advances in Clinical and Experimental Medicine. https://doi.org/10.17219/acem/216070
Vancouver
Lettner J, Ostojic M, Królikowska A, Kayaalp ME, Ramadanov N, Prill R. A new peer reviewer? Comparing AI with human performance in randomized controlled trial risk-of-bias assessment. Advances in Clinical and Experimental Medicine. 2026. doi:10.17219/acem/216070
BibTeX
@article{Lettner2026,
author = {Lettner, Jonathan and Ostojic, Marko and Królikowska, Aleksandra and Kayaalp, Mahmut Enes and Ramadanov, Nikolai and Prill, Robert},
title = {A new peer reviewer? Comparing AI with human performance in randomized controlled trial risk-of-bias assessment},
journal = {Advances in Clinical and Experimental Medicine},
year = {2026},
doi = {10.17219/acem/216070}
}
EUVIMED