EUVIMEDEUVIMED entdecken
Systematische ÜbersichtOpen Access · CC BY-NC 4.0

A real‐world evaluation of the implementation of NLP technology in abstract screening of a systematic review

Sara Perlman-Arrow iD, Noel Loo, Niklas Bobrovitz iD, Tingting Yan, Rahul K Arora

Research Synthesis Methods · 2023-07 · 14 (4) · 608-621

DOI: 10.1002/jrsm.1636 ↗

Kurz gefasst

Systematische Übersicht in der Zeitschrift Research Synthesis Methods, online erschienen am 25.05.2023. Verfasst von Sara Perlman-Arrow und 4 weiteren Autor:innen. Beteiligte Einrichtung: School of Population and Global Health, McGill University, Quebec, Canada. Hauptthemen laut MEDLINE: Natural Language Processing, COVID-19. Die Arbeit wurde bisher 29-mal zitiert (Crossref) und nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: World Health Organization, Public Health Agency of Canada, Rhodes Scholarships.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The laborious and time‐consuming nature of systematic review production hinders the dissemination of up‐to‐date evidence synthesis. Well‐performing natural language processing (NLP) tools for systematic reviews have been developed, showing promise to improve efficiency. However, the feasibility and value of these technologies have not been comprehensively demonstrated in a real‐world review. We developed an NLP‐assisted abstract screening tool that provides text inclusion recommendations, keyword highlights, and visual context cues. We evaluated this tool in a living systematic review on SARS‐CoV‐2 seroprevalence, conducting a quality improvement assessment of screening with and without the tool. We evaluated changes to abstract screening speed, screening accuracy, characteristics of included texts, and user satisfaction. The tool improved efficiency, reducing screening time per abstract by 45.9% and decreasing inter‐reviewer conflict rates. The tool conserved precision of article inclusion (positive predictive value; 0.92 with tool vs. 0.88 without) and recall (sensitivity; 0.90 vs. 0.81). The summary statistics of included studies were similar with and without the tool. Users were satisfied with the tool (mean satisfaction score of 4.2/5). We evaluated an abstract screening process where one human reviewer was replaced with the tool's votes, finding that this maintained recall (0.92 one‐person, one‐tool vs. 0.90 two tool‐assisted humans) and precision (0.91 vs. 0.92) while reducing screening time by 70%. Implementing an NLP tool in this living systematic review improved efficiency, maintained accuracy, and was well‐received by researchers, demonstrating the real‐world effectiveness of NLP in expediting evidence synthesis.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language ProcessingCOVID-19HumansSeroepidemiologic StudiesSARS-CoV-2Systematic Reviews as Topic

Stichwörter der Autor:innen

abstract screeningliving literature reviewnatural language processingsystematic reviewtext classification

Autor:innen und Institutionen

  • School of Population and Global Health, McGill University, Quebec, Canada.
  • Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA.
  • Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada.
  • Centre for Health Informatics, University of Calgary, Calgary, Alberta, Canada.
  • Institute of Biomedical Engineering, University of Oxford, Oxford, UK.

ORCID-Profile: Sara Perlman-Arrow, Niklas Bobrovitz

Förderung

  • World Health Organization
  • Public Health Agency of Canada
  • Rhodes Scholarships
  • Robert Koch Institut

Zitieren

APA

Perlman-Arrow, S., Loo, N., Bobrovitz, N., Yan, T., & Arora, R. K. (2023). A real‐world evaluation of the implementation of NLP technology in abstract screening of a systematic review. Research Synthesis Methods, 14(4), 608-621. https://doi.org/10.1002/jrsm.1636

Vancouver

Perlman-Arrow S, Loo N, Bobrovitz N, Yan T, Arora RK. A real‐world evaluation of the implementation of NLP technology in abstract screening of a systematic review. Research Synthesis Methods. 2023;14(4):608-621. doi:10.1002/jrsm.1636

BibTeX
@article{PerlmanArrow2023,
  author = {Perlman-Arrow, Sara and Loo, Noel and Bobrovitz, Niklas and Yan, Tingting and Arora, Rahul K},
  title = {A real‐world evaluation of the implementation of NLP technology in abstract screening of a systematic review},
  journal = {Research Synthesis Methods},
  year = {2023},
  volume = {14},
  number = {4},
  pages = {608--621},
  doi = {10.1002/jrsm.1636}
}
Kostenlos starten →