High-performance automated abstract screening with large language model ensembles
Journal of the American Medical Informatics Association · 01.05.2025 · 32 (5) · 893-904
DOI: 10.1093/jamia/ocaf050 ↗Kurz gefasst
Validierungsstudie in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 22.03.2025. Verfasst von Rohan Sanghera und 9 weiteren Autor:innen. Beteiligte Einrichtung: Oxford University Hospitals NHS Foundation Trust, Oxford OX3 9DU, United Kingdom. Hauptthemen laut MEDLINE: Natural Language Processing, Systematic Reviews as Topic, Abstracting and Indexing. Die Arbeit wurde bisher 71-mal zitiert (Crossref) und nennt 51 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: HealthSense Research Fund, Microsoft Research Accelerating Foundation, Microsoft Research Accelerating Foundation Models Research.
Abstract (Originalsprache)
Objective
Abstract screening is a labor-intensive component of systematic review involving repetitive application of inclusion and exclusion criteria on a large volume of studies. We aimed to validate large language models (LLMs) used to automate abstract screening. Materials and Methods LLMs (GPT-3.5 Turbo, GPT-4 Turbo, GPT-4o, Llama 3 70B, Gemini 1.5 Pro, and Claude Sonnet 3.5) were trialed across 23 Cochrane Library systematic reviews to evaluate their accuracy in zero-shot binary classification for abstract screening. Initial evaluation on a balanced development dataset (n = 800) identified optimal prompting strategies, and the best performing LLM-prompt combinations were then validated on a comprehensive dataset of replicated search results (n = 119 695).
Results
On the development dataset, LLMs exhibited superior performance to human researchers in terms of sensitivity (LLMmax = 1.000, humanmax = 0.775), precision (LLMmax = 0.927, humanmax = 0.911), and balanced accuracy (LLMmax = 0.904, humanmax = 0.865). When evaluated on the comprehensive dataset, the best performing LLM-prompt combinations exhibited consistent sensitivity (range 0.756-1.000) but diminished precision (range 0.004-0.096) due to class imbalance. In addition, 66 LLM-human and LLM-LLM ensembles exhibited perfect sensitivity with a maximal precision of 0.458 with the development dataset, decreasing to 0.1450 over the comprehensive dataset; but conferring workload reductions ranging between 37.55% and 99.11%.
Discussion
Automated abstract screening can reduce the screening workload in systematic review while maintaining quality. Performance variation between reviews highlights the importance of domain-specific validation before autonomous deployment. LLM-human ensembles can achieve similar benefits while maintaining human oversight over all records.
Conclusion
LLMs may reduce the human labor cost of systematic review with maintained or improved accuracy, thereby increasing the efficiency and quality of evidence synthesis.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Oxford University Hospitals NHS Foundation Trust, Oxford OX3 9DU, United Kingdom.
- Oxford University Clinical Academic Graduate School, Medical Sciences Division, University of Oxford, Oxford OX3 9DU, United Kingdom.
- Nuffield Department of Clinical Neurosciences, Medical Sciences Division, University of Oxford, Oxford OX3 9DU, United Kingdom.
- School of Clinical Medicine, University of Cambridge, Cambridge CB2 0SP, United Kingdom.
- Georgetown University School of Medicine, Georgetown University, Washington, DC 20007, United States.
- MedStar Washington Hospital Center, Washington, DC 20010, United States.
ORCID-Profile: Arun James Thirunavukarasu, Andrew A S Soltan
Förderung
- HealthSense Research Fund
- Microsoft Research Accelerating Foundation
- Microsoft Research Accelerating Foundation Models Research
- National Institute for Health and Care Research
- NHS Foundation Trust
- NHS
- Department of Health and Social Care
Zitieren
APA
Sanghera, R., Thirunavukarasu, A. J., El Khoury, M., O'Logbon, J., Chen, Y., Watt, A., Mahmood, M., Butt, H., Nishimura, G., & Soltan, A. A. S. (2025). High-performance automated abstract screening with large language model ensembles. Journal of the American Medical Informatics Association, 32(5), 893-904. https://doi.org/10.1093/jamia/ocaf050
Vancouver
Sanghera R, Thirunavukarasu AJ, El Khoury M, O'Logbon J, Chen Y, Watt A, et al. High-performance automated abstract screening with large language model ensembles. Journal of the American Medical Informatics Association. 2025;32(5):893-904. doi:10.1093/jamia/ocaf050
BibTeX
@article{Sanghera2025,
author = {Sanghera, Rohan and Thirunavukarasu, Arun James and El Khoury, Marc and O'Logbon, Jessica and Chen, Yuqing and Watt, Archie and Mahmood, Mustafa and Butt, Hamid and Nishimura, George and Soltan, Andrew A S},
title = {High-performance automated abstract screening with large language model ensembles},
journal = {Journal of the American Medical Informatics Association},
year = {2025},
volume = {32},
number = {5},
pages = {893--904},
doi = {10.1093/jamia/ocaf050}
}
EUVIMED