EUVIMEDEUVIMED entdecken
ValidierungsstudieOpen Access

A Fine-Tuned Large Language Model Workflow to Support Risk of Bias Assessment: Development, External Validation, and Reviewer Verification Study

Yitian Li, Jingyu Chen, Liting Liang, Xingyun Geng, Guohua Wang, Jinsong Geng

Journal of Medical Internet Research · 13.11.2026 · 28 · e108001-e108001

DOI: 10.2196/108001 ↗

Kurz gefasst

Validierungsstudie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 13.11.2026. Verfasst von Yitian Li und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Medical Informatics, Medical School of Nantong University, Nantong, Jiangsu, China. Hauptthemen laut MEDLINE: Workflow, Randomized Controlled Trials as Topic. Die Arbeit nennt 42 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Given that randomized controlled trials (RCTs) are the gold standard for effectiveness research, assessing their risk of bias (RoB) becomes an indispensable step in systematic reviews. Yet this task remains time-consuming and prone to reviewer subjectivity. While large language models may accelerate the process, they lack transparent, traceable judgments linked to trial reports.

Objective

To bridge methodological gaps and improve both efficiency and consistency, this study developed and externally validated TRoB-1, a fine-tuned open-weight large language model workflow for RoB assessment of RCTs.

Methods

TRoB-1 was developed specifically for the Cochrane risk of bias tool version 1 (RoB 1), which is the most widely adopted instrument for critical appraisal of RCTs. The workflow generates preliminary judgments and supporting quotations from reviewer-selected passages for the target RoB domains. The workflow was built on 2-stage supervised fine-tuning of Gemma 2-9B-Instruct using 4-bit quantized low-rank adaptation. The training set comprised 4321 RoB records extracted from systematic reviews of chronic obstructive pulmonary disease, diabetes, and hypertension. The validation set consisted of 161 records from recently published RCTs covering a broader disease area. The reference standard was established by 2 independent reviewers. Two clinicians blinded to the reference standard verified TRoB-1-assisted assessments through a locally hosted web interface. Using the same reference standard, the performance of TRoB-1 raw outputs was compared with that of the Gemma 2 base workflow, and clinician-verified outputs were compared with raw TRoB-1 outputs.

Results

TRoB-1 successfully generated structured outputs for all 161 validation records. In independent clinician-assisted assessments using the same workflow, raw TRoB-1 accuracy ranged from 73.3% (n=118) to 73.9% (n=119), compared with 52.2% (n=84) to 60.2% (n=97) for the Gemma 2 base model. After verification, accuracy ranged from 82.0% (n=132) to 84.5% (n=136) and macro- F 1 from 0.709 to 0.776. Accuracy remained higher for raw TRoB-1 after excluding invalid base-model outputs, and verification did not change any correct judgment to an incorrect one. Clinician 1 corrected 13 initially incorrect judgments and clinician 2 corrected 18. The F 1 -score for judgments of high-RoB increased from 0.600 to 0.784 for clinician 1 and from 0.571 to 0.821 for clinician 2.

Conclusions

TRoB-1 combined task-specific fine-tuning, evidence-linked outputs, and clinician verification in a transparent workflow for reviewer-supported RoB assessment. The TRoB-1 web interface provides a practical platform for integrating RoB into evidence synthesis, particularly for living systematic reviews where timely updates are critical.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

WorkflowRandomized Controlled Trials as TopicLarge Language ModelsBiasHumansRisk Assessment

Stichwörter der Autor:innen

large language modelrandomized controlled trialrisk of biasvalidationworkflow

Autor:innen und Institutionen

  • Department of Medical Informatics, Medical School of Nantong University, Nantong, Jiangsu, China.
  • Center for Evidence-Based Medicine, Medical School of Nantong University, 19 Qixiu Road, Medical School of Nantong University, Nantong, Jiangsu, 226001, China, 86 51385051891.
  • National Stroke Administration Office, Xuanwu Hospital Capital Medical University, Beijing, China.

Zitieren

APA

Li, Y., Chen, J., Liang, L., Geng, X., Wang, G., & Geng, J. (2026). A Fine-Tuned Large Language Model Workflow to Support Risk of Bias Assessment: Development, External Validation, and Reviewer Verification Study. Journal of Medical Internet Research, 28, e108001-e108001. https://doi.org/10.2196/108001

Vancouver

Li Y, Chen J, Liang L, Geng X, Wang G, Geng J. A Fine-Tuned Large Language Model Workflow to Support Risk of Bias Assessment: Development, External Validation, and Reviewer Verification Study. Journal of Medical Internet Research. 2026;28:e108001-e108001. doi:10.2196/108001

BibTeX
@article{Li2026,
  author = {Li, Yitian and Chen, Jingyu and Liang, Liting and Geng, Xingyun and Wang, Guohua and Geng, Jinsong},
  title = {A Fine-Tuned Large Language Model Workflow to Support Risk of Bias Assessment: Development, External Validation, and Reviewer Verification Study},
  journal = {Journal of Medical Internet Research},
  year = {2026},
  volume = {28},
  pages = {e108001--e108001},
  doi = {10.2196/108001}
}
Kostenlos starten →