A comprehensive systematic review dataset is a rich resource for training and evaluation of AI systems for title and abstract screening
Research Synthesis Methods · 2025
DOI: 10.1017/rsm.2025.1 ↗Kurz gefasst
Systematische Übersicht in der Zeitschrift Research Synthesis Methods, 2025. Verfasst von Gary C K Chan und 3 weiteren Autor:innen. Beteiligte Einrichtung: School of Computing Technologies, RMIT University, Melbourne, VIC, Australia. Hauptthemen laut MEDLINE: Artificial Intelligence, Systematic Reviews as Topic. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 27 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
When conducting a systematic review, screening the vast body of literature to identify the small set of relevant studies is a labour-intensive and error-prone process. Although there is an increasing number of fully automated tools for screening, their performance is suboptimal and varies substantially across review topic areas. Many of these tools are only trained on small datasets, and most are not tested on a wide range of review topic areas. This study presents two systematic review datasets compiled from more than 8600 systematic reviews and more than 540000 abstracts covering 51 research topic areas in health and medical research. These datasets are the largest of their kinds to date. We demonstrate their utility in training and evaluating language models for title and abstract screening. Our dataset includes detailed metadata of each review, including title, background, objectives and selection criteria. We demonstrated that a small language model trained on this dataset with additional metadata has excellent performance with an average recall above 95% and specificity over 70% across a wide range of review topic areas. Future research can build on our dataset to further improve the performance of fully automated tools for systematic review title and abstract screening.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- School of Computing Technologies, RMIT University, Melbourne, VIC, Australia.
- National Centre for Youth Substance Use Research, University of Queensland, Brisbane, QLD, Australia.
- School of Computing and Information Systems, The University of Melbourne, Melbourne, VIC, Australia.
Zitieren
APA
Chan, G. C. K., He, E., Leung, J., & Verspoor, K. (2025). A comprehensive systematic review dataset is a rich resource for training and evaluation of AI systems for title and abstract screening. Research Synthesis Methods. https://doi.org/10.1017/rsm.2025.1
Vancouver
Chan GCK, He E, Leung J, Verspoor K. A comprehensive systematic review dataset is a rich resource for training and evaluation of AI systems for title and abstract screening. Research Synthesis Methods. 2025. doi:10.1017/rsm.2025.1
BibTeX
@article{Chan2025,
author = {Chan, Gary C K and He, Estrid and Leung, Janni and Verspoor, Karin},
title = {A comprehensive systematic review dataset is a rich resource for training and evaluation of AI systems for title and abstract screening},
journal = {Research Synthesis Methods},
year = {2025},
doi = {10.1017/rsm.2025.1}
}
EUVIMED