A real‐world evaluation of the implementation of NLP technology in abstract screening of a systematic review
Research Synthesis Methods · 2023-07 · 14 (4) · 608-621
DOI: 10.1002/jrsm.1636 ↗Kurz gefasst
Systematische Übersicht in der Zeitschrift Research Synthesis Methods, online erschienen am 25.05.2023. Verfasst von Sara Perlman-Arrow und 4 weiteren Autor:innen. Beteiligte Einrichtung: School of Population and Global Health, McGill University, Quebec, Canada. Hauptthemen laut MEDLINE: Natural Language Processing, COVID-19. Die Arbeit wurde bisher 29-mal zitiert (Crossref) und nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: World Health Organization, Public Health Agency of Canada, Rhodes Scholarships.
Abstract (Originalsprache)
The laborious and time‐consuming nature of systematic review production hinders the dissemination of up‐to‐date evidence synthesis. Well‐performing natural language processing (NLP) tools for systematic reviews have been developed, showing promise to improve efficiency. However, the feasibility and value of these technologies have not been comprehensively demonstrated in a real‐world review. We developed an NLP‐assisted abstract screening tool that provides text inclusion recommendations, keyword highlights, and visual context cues. We evaluated this tool in a living systematic review on SARS‐CoV‐2 seroprevalence, conducting a quality improvement assessment of screening with and without the tool. We evaluated changes to abstract screening speed, screening accuracy, characteristics of included texts, and user satisfaction. The tool improved efficiency, reducing screening time per abstract by 45.9% and decreasing inter‐reviewer conflict rates. The tool conserved precision of article inclusion (positive predictive value; 0.92 with tool vs. 0.88 without) and recall (sensitivity; 0.90 vs. 0.81). The summary statistics of included studies were similar with and without the tool. Users were satisfied with the tool (mean satisfaction score of 4.2/5). We evaluated an abstract screening process where one human reviewer was replaced with the tool's votes, finding that this maintained recall (0.92 one‐person, one‐tool vs. 0.90 two tool‐assisted humans) and precision (0.91 vs. 0.92) while reducing screening time by 70%. Implementing an NLP tool in this living systematic review improved efficiency, maintained accuracy, and was well‐received by researchers, demonstrating the real‐world effectiveness of NLP in expediting evidence synthesis.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- School of Population and Global Health, McGill University, Quebec, Canada.
- Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA.
- Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada.
- Centre for Health Informatics, University of Calgary, Calgary, Alberta, Canada.
- Institute of Biomedical Engineering, University of Oxford, Oxford, UK.
ORCID-Profile: Sara Perlman-Arrow, Niklas Bobrovitz
Förderung
- World Health Organization
- Public Health Agency of Canada
- Rhodes Scholarships
- Robert Koch Institut
Zitieren
APA
Perlman-Arrow, S., Loo, N., Bobrovitz, N., Yan, T., & Arora, R. K. (2023). A real‐world evaluation of the implementation of NLP technology in abstract screening of a systematic review. Research Synthesis Methods, 14(4), 608-621. https://doi.org/10.1002/jrsm.1636
Vancouver
Perlman-Arrow S, Loo N, Bobrovitz N, Yan T, Arora RK. A real‐world evaluation of the implementation of NLP technology in abstract screening of a systematic review. Research Synthesis Methods. 2023;14(4):608-621. doi:10.1002/jrsm.1636
BibTeX
@article{PerlmanArrow2023,
author = {Perlman-Arrow, Sara and Loo, Noel and Bobrovitz, Niklas and Yan, Tingting and Arora, Rahul K},
title = {A real‐world evaluation of the implementation of NLP technology in abstract screening of a systematic review},
journal = {Research Synthesis Methods},
year = {2023},
volume = {14},
number = {4},
pages = {608--621},
doi = {10.1002/jrsm.1636}
}
EUVIMED