EUVIMEDEUVIMED entdecken
ValidierungsstudieOpen Access · CC BY 4.0

High-performance automated abstract screening with large language model ensembles

Rohan Sanghera, Arun James Thirunavukarasu iD, Marc El Khoury, Jessica O'Logbon, Yuqing Chen, Archie Watt, Mustafa Mahmood, Hamid Butt, George Nishimura, Andrew A S Soltan iD

Journal of the American Medical Informatics Association · 01.05.2025 · 32 (5) · 893-904

DOI: 10.1093/jamia/ocaf050 ↗

Kurz gefasst

Validierungsstudie in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 22.03.2025. Verfasst von Rohan Sanghera und 9 weiteren Autor:innen. Beteiligte Einrichtung: Oxford University Hospitals NHS Foundation Trust, Oxford OX3 9DU, United Kingdom. Hauptthemen laut MEDLINE: Natural Language Processing, Systematic Reviews as Topic, Abstracting and Indexing. Die Arbeit wurde bisher 71-mal zitiert (Crossref) und nennt 51 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: HealthSense Research Fund, Microsoft Research Accelerating Foundation, Microsoft Research Accelerating Foundation Models Research.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

Abstract screening is a labor-intensive component of systematic review involving repetitive application of inclusion and exclusion criteria on a large volume of studies. We aimed to validate large language models (LLMs) used to automate abstract screening. Materials and Methods LLMs (GPT-3.5 Turbo, GPT-4 Turbo, GPT-4o, Llama 3 70B, Gemini 1.5 Pro, and Claude Sonnet 3.5) were trialed across 23 Cochrane Library systematic reviews to evaluate their accuracy in zero-shot binary classification for abstract screening. Initial evaluation on a balanced development dataset (n = 800) identified optimal prompting strategies, and the best performing LLM-prompt combinations were then validated on a comprehensive dataset of replicated search results (n = 119 695).

Results

On the development dataset, LLMs exhibited superior performance to human researchers in terms of sensitivity (LLMmax = 1.000, humanmax = 0.775), precision (LLMmax = 0.927, humanmax = 0.911), and balanced accuracy (LLMmax = 0.904, humanmax = 0.865). When evaluated on the comprehensive dataset, the best performing LLM-prompt combinations exhibited consistent sensitivity (range 0.756-1.000) but diminished precision (range 0.004-0.096) due to class imbalance. In addition, 66 LLM-human and LLM-LLM ensembles exhibited perfect sensitivity with a maximal precision of 0.458 with the development dataset, decreasing to 0.1450 over the comprehensive dataset; but conferring workload reductions ranging between 37.55% and 99.11%.

Discussion

Automated abstract screening can reduce the screening workload in systematic review while maintaining quality. Performance variation between reviews highlights the importance of domain-specific validation before autonomous deployment. LLM-human ensembles can achieve similar benefits while maintaining human oversight over all records.

Conclusion

LLMs may reduce the human labor cost of systematic review with maintained or improved accuracy, thereby increasing the efficiency and quality of evidence synthesis.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language ProcessingSystematic Reviews as TopicAbstracting and IndexingHumansLarge Language Models

Stichwörter der Autor:innen

abstract screeningartificial intelligenceevidence synthesisfoundation modellarge language modelsystematic review

Autor:innen und Institutionen

  • Oxford University Hospitals NHS Foundation Trust, Oxford OX3 9DU, United Kingdom.
  • Oxford University Clinical Academic Graduate School, Medical Sciences Division, University of Oxford, Oxford OX3 9DU, United Kingdom.
  • Nuffield Department of Clinical Neurosciences, Medical Sciences Division, University of Oxford, Oxford OX3 9DU, United Kingdom.
  • School of Clinical Medicine, University of Cambridge, Cambridge CB2 0SP, United Kingdom.
  • Georgetown University School of Medicine, Georgetown University, Washington, DC 20007, United States.
  • MedStar Washington Hospital Center, Washington, DC 20010, United States.

ORCID-Profile: Arun James Thirunavukarasu, Andrew A S Soltan

Förderung

  • HealthSense Research Fund
  • Microsoft Research Accelerating Foundation
  • Microsoft Research Accelerating Foundation Models Research
  • National Institute for Health and Care Research
  • NHS Foundation Trust
  • NHS
  • Department of Health and Social Care

Zitieren

APA

Sanghera, R., Thirunavukarasu, A. J., El Khoury, M., O'Logbon, J., Chen, Y., Watt, A., Mahmood, M., Butt, H., Nishimura, G., & Soltan, A. A. S. (2025). High-performance automated abstract screening with large language model ensembles. Journal of the American Medical Informatics Association, 32(5), 893-904. https://doi.org/10.1093/jamia/ocaf050

Vancouver

Sanghera R, Thirunavukarasu AJ, El Khoury M, O'Logbon J, Chen Y, Watt A, et al. High-performance automated abstract screening with large language model ensembles. Journal of the American Medical Informatics Association. 2025;32(5):893-904. doi:10.1093/jamia/ocaf050

BibTeX
@article{Sanghera2025,
  author = {Sanghera, Rohan and Thirunavukarasu, Arun James and El Khoury, Marc and O'Logbon, Jessica and Chen, Yuqing and Watt, Archie and Mahmood, Mustafa and Butt, Hamid and Nishimura, George and Soltan, Andrew A S},
  title = {High-performance automated abstract screening with large language model ensembles},
  journal = {Journal of the American Medical Informatics Association},
  year = {2025},
  volume = {32},
  number = {5},
  pages = {893--904},
  doi = {10.1093/jamia/ocaf050}
}
Kostenlos starten →