EUVIMEDEUVIMED entdecken
Crossref

Assessing the Reliability of Large Language Models for Evaluation of Risk-of-Bias in Randomized Clinical Trials

Takeshi Nagao, Tetsuya Kawakita iD

American Journal of Perinatology · 2026-08 · 43 (11) · 1527-1532

DOI: 10.1055/a-2793-9092 ↗

Kurz gefasst

Fachartikel in der Zeitschrift American Journal of Perinatology, online erschienen am 05.03.2026. Verfasst von Takeshi Nagao und 1 weiteren Autor:innen. Beteiligte Einrichtung: Department of Obstetrics and Gynecology, The Jikei University School of Medicine, Tokyo, Japan. Hauptthemen laut MEDLINE: Large Language Models, Randomized Controlled Trials as Topic.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

Systematic reviews depend on rigorous risk-of-bias (RoB) assessments to ensure credibility, yet manual evaluation using the Cochrane RoB 2 tool is resource-intensive. While Large Language Models (LLMs) offer potential for automation, their alignment with human judgment remains underexplored. This study evaluates the reliability of ChatGPT-4o, ChatGPT-5, and Claude 3.5 Sonnet in assessing RoB in randomized controlled trials (RCTs), comparing their agreement with human reviewers and internal consistency. Study Design We retrospectively analyzed 180 RCTs from systematic reviews published in the American Journal of Obstetrics and Gynecology (2021–2023) reporting complete human RoB 2 ratings. Each LLM processed full-text PDFs using a standardized prompt incorporating the complete RoB 2 algorithm. Model performance was evaluated against human benchmarks using Cohen’s kappa and prevalence- and bias-adjusted kappa (PABAK). Intra-model reliability was assessed across three independent runs to measure consistency.

Results

ChatGPT-5 consistently outperformed other models, achieving the highest agreement in randomization (Domain 1; 76%), missing outcome data (Domain 3; 80%), and outcome measurement (Domain 4; 76%). It showed moderate concordance for deviations from intended interventions (69%). However, all models struggled with selective reporting (Domain 5), where agreement dropped to 47–51%. For overall risk-of-bias judgments, ChatGPT-5 demonstrated superior concordance (60–62%, κ=0.36–0.40) compared to ChatGPT-4o (45%) and Claude 3.5 Sonnet (43%). ChatGPT-5 also exhibited substantial to near-perfect internal consistency.

Conclusion

Among the evaluated models, ChatGPT-5 most closely approximated human RoB 2 assessments and achieved superior internal consistency, suggesting it could serve as a practical first-pass tool to reduce reviewer burden. However, persistent limitations in detecting selective reporting—likely due to the inability to cross-reference external trial registries—highlight that expert human oversight remains essential for accurate evidence synthesis.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsRandomized Controlled Trials as TopicHumansReproducibility of ResultsBiasRetrospective Studies

Autor:innen und Institutionen

  • Department of Obstetrics and Gynecology, The Jikei University School of Medicine, Tokyo, Japan.
  • Department of Obstetrics and Gynecology, Macon and Joan Brock Virginia Health Sciences Eastern Virginia Medical School at Old Dominion University, Norfolk, Virginia, United States.

ORCID-Profile: Tetsuya Kawakita

Zitieren

APA

Nagao, T., & Kawakita, T. (2026). Assessing the Reliability of Large Language Models for Evaluation of Risk-of-Bias in Randomized Clinical Trials. American Journal of Perinatology, 43(11), 1527-1532. https://doi.org/10.1055/a-2793-9092

Vancouver

Nagao T, Kawakita T. Assessing the Reliability of Large Language Models for Evaluation of Risk-of-Bias in Randomized Clinical Trials. American Journal of Perinatology. 2026;43(11):1527-1532. doi:10.1055/a-2793-9092

BibTeX
@article{Nagao2026,
  author = {Nagao, Takeshi and Kawakita, Tetsuya},
  title = {Assessing the Reliability of Large Language Models for Evaluation of Risk-of-Bias in Randomized Clinical Trials},
  journal = {American Journal of Perinatology},
  year = {2026},
  volume = {43},
  number = {11},
  pages = {1527--1532},
  doi = {10.1055/a-2793-9092}
}
Kostenlos starten →