EUVIMEDEUVIMED entdecken
Open Access

RoBuster: A Corpus Annotated with Risk of Bias Text Spans in Randomized Controlled Trials in Physiotherapy and Rehabilitation (Preprint)

Anjani Dhrangadhariya iD, Roger Hilfiker iD, Karl Martin Sattelmayer iD, Nona Naderi iD, Katia Giacomino iD, Rahel Caliesch iD, Julian Higgins iD, Stéphane Marchand-Maillet iD, Henning Müller iD

JMIR Formative Research · 2023

DOI: 10.2196/55127 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Formative Research, 2023. Verfasst von Anjani Dhrangadhariya und 8 weiteren Autor:innen. Beteiligte Einrichtung: Informatics Institute, HES-SO Valais-Wallis, Rue du Technopole 3, Sierre, 3960, Switzerland, 41 787084007. Hauptthemen laut MEDLINE: Randomized Controlled Trials as Topic, Physical Therapy Modalities, Data Curation. Die Arbeit nennt 1 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Risk of bias (RoB) assessment of randomized clinical trials (RCTs) is vital to answering systematic review questions accurately. Manual RoB assessment for hundreds of RCTs is a cognitively demanding and lengthy process. Automation has the potential to assist reviewers in rapidly identifying text descriptions in RCTs that indicate potential risks of bias. However, no RoB text span annotated corpus could be used to fine-tune or evaluate large language models (LLMs), and there are no established guidelines for annotating the RoB spans in RCTs.

Objective

The revised Cochrane RoB 2 test (RoB 2) tool provides comprehensive guidelines for RoB assessment; however, due to the inherent subjectivity of this tool, it cannot be directly used as RoB annotation guidelines. The study aimed to develop precise RoB text span annotation instructions that could address this subjectivity and thus aid the corpus annotation.

Methods

We leveraged RoB 2 guidelines to develop visual instructional placards that serve as annotation guidelines for RoB spans and risk judgments. Expert annotators used these visual placards to annotate a dataset named RoBuster, consisting of 41 full-text RCTs from the domains of physiotherapy and rehabilitation. We report interannotator agreement (IAA) between 2 annotators for text span annotations before and after applying visual instructions on a subset (n=9) of RoBuster. We also provide IAA on bias risk judgments using Cohen κ. Moreover, we used a portion of RoBuster (n=10) to evaluate an LLM using a straightforward evaluation framework. This evaluation aimed to gauge the performance of an LLM (here GPT 3.5) in the challenging task of RoB span extraction and demonstrate the utility of this corpus using a straightforward framework.

Results

We present a corpus of 41 RCTs with fine-grained text span annotations comprising more than 28,427 tokens belonging to 22 RoB classes. The IAA at the text span level calculated using the F1 measure varies from 0% to 90%, while Cohen κ for risk judgments ranges between -0.235 and 1.0. Using visual instructions for annotation increases the IAA by more than 17 percentage points. LLM (GPT-3.5) shows promising but varied observed agreements with the expert annotation across the different bias questions.

Conclusions

Despite having comprehensive bias assessment guidelines and visual instructional placards, RoB annotation remains a complex task. Using visual placards for bias assessment and annotation enhances IAA compared to cases where visual placards are absent; however, text annotation remains challenging for the subjective questions and the questions for which annotation data are unavailable in RCTs. Similarly, while GPT-3.5 demonstrates effectiveness, its accuracy diminishes with more subjective RoB questions and low information availability.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Randomized Controlled Trials as TopicPhysical Therapy ModalitiesData CurationBiasHumansLarge Language Models

Stichwörter der Autor:innen

LLMRCTRoBustercorpuscorpus annotationeffectivenessinformation extractionlarge language modelsnatural language processingphysiotherapyrandomized controlled trialsrehabilitationreviewerrisk of biastools

Autor:innen und Institutionen

  • Informatics Institute, HES-SO Valais-Wallis, Rue du Technopole 3, Sierre, 3960, Switzerland, 41 787084007.
  • Institute of Health Sciences, HES-SO Valais-Wallis, Thermenstrasse 41, Leukerbad, Valais, Switzerland.
  • Physiotherapy Tschopp & Hilfiker, Brig, Valais, Switzerland.
  • Centre national de la recherche scientifique, Laboratoire Interdisciplinaire des Sciences du Numérique, Université Paris-Saclay, Orsay, France.
  • Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, United Kingdom.
  • Centre d'Informatique Universitaire, University of Geneva, Geneva, Switzerland.

ORCID-Profile: Anjani Dhrangadhariya, Roger Hilfiker, Karl Martin Sattelmayer, Nona Naderi, Katia Giacomino, Rahel Caliesch, Julian Higgins, Stéphane Marchand-Maillet, Henning Müller

Zitieren

APA

Dhrangadhariya, A., Hilfiker, R., Sattelmayer, K. M., Naderi, N., Giacomino, K., Caliesch, R., Higgins, J., Marchand-Maillet, S., & Müller, H. (2023). RoBuster: A Corpus Annotated with Risk of Bias Text Spans in Randomized Controlled Trials in Physiotherapy and Rehabilitation (Preprint). JMIR Formative Research. https://doi.org/10.2196/55127

Vancouver

Dhrangadhariya A, Hilfiker R, Sattelmayer KM, Naderi N, Giacomino K, Caliesch R, et al. RoBuster: A Corpus Annotated with Risk of Bias Text Spans in Randomized Controlled Trials in Physiotherapy and Rehabilitation (Preprint). JMIR Formative Research. 2023. doi:10.2196/55127

BibTeX
@article{Dhrangadhariya2023,
  author = {Dhrangadhariya, Anjani and Hilfiker, Roger and Sattelmayer, Karl Martin and Naderi, Nona and Giacomino, Katia and Caliesch, Rahel and Higgins, Julian and Marchand-Maillet, Stéphane and Müller, Henning},
  title = {RoBuster: A Corpus Annotated with Risk of Bias Text Spans in Randomized Controlled Trials in Physiotherapy and Rehabilitation (Preprint)},
  journal = {JMIR Formative Research},
  year = {2023},
  doi = {10.2196/55127}
}
Kostenlos starten →