A question-answering framework for automated abstract screening using large language models
Journal of the American Medical Informatics Association · 01.09.2024 · 31 (9) · 1939-1952
DOI: 10.1093/jamia/ocae166 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 23.07.2024. Verfasst von Opeoluwa Akinseloyin und 2 weiteren Autor:innen. Beteiligte Einrichtung: Centre for Computational Science and Mathematical Modelling, Coventry University, Coventry CV1 2TT, United Kingdom. Hauptthemen laut MEDLINE: Natural Language Processing, Artificial Intelligence. Die Arbeit wurde bisher 36-mal zitiert (Crossref) und nennt 73 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: National Planning Office of Philosophy and Social Science of China, International Exchange Scheme of the Royal Society of the United Kingdom, Research Excellence Development Framework award of Coventry University.
Abstract (Originalsprache)
Objective
This paper aims to address the challenges in abstract screening within systematic reviews (SR) by leveraging the zero-shot capabilities of large language models (LLMs).
Methods
We employ LLM to prioritize candidate studies by aligning abstracts with the selection criteria outlined in an SR protocol. Abstract screening was transformed into a novel question-answering (QA) framework, treating each selection criterion as a question addressed by LLM. The framework involves breaking down the selection criteria into multiple questions, properly prompting LLM to answer each question, scoring and re-ranking each answer, and combining the responses to make nuanced inclusion or exclusion decisions. Results and Discussion Large-scale validation was performed on the benchmark of CLEF eHealth 2019 Task 2: Technology-Assisted Reviews in Empirical Medicine. Focusing on GPT-3.5 as a case study, the proposed QA framework consistently exhibited a clear advantage over traditional information retrieval approaches and bespoke BERT-family models that were fine-tuned for prioritizing candidate studies (ie, from the BERT to PubMedBERT) across 31 datasets of 4 categories of SRs, underscoring their high potential in facilitating abstract screening. The experiments also showcased the viability of using selection criteria as a query for reference prioritization. The experiments also showcased the viability of the framework using different LLMs.
Conclusion
Investigation justified the indispensable value of leveraging selection criteria to improve the performance of automated abstract screening. LLMs demonstrated proficiency in prioritizing candidate studies for abstract screening using the proposed QA framework. Significant performance improvements were obtained by re-ranking answers using the semantic alignment between abstracts and selection criteria. This further highlighted the pertinence of utilizing selection criteria to enhance abstract screening.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Centre for Computational Science and Mathematical Modelling, Coventry University, Coventry CV1 2TT, United Kingdom.
- Information School, The University of Sheffield, Sheffield S10 2AH, United Kingdom.
ORCID-Profile: Xiaorui Jiang, Vasile Palade
Förderung
- National Planning Office of Philosophy and Social Science of China
- International Exchange Scheme of the Royal Society of the United Kingdom
- Research Excellence Development Framework award of Coventry University
- Coventry University
- International Exchange Scheme
- Royal Society of the United Kingdom
Zitieren
APA
Akinseloyin, O., Jiang, X., & Palade, V. (2024). A question-answering framework for automated abstract screening using large language models. Journal of the American Medical Informatics Association, 31(9), 1939-1952. https://doi.org/10.1093/jamia/ocae166
Vancouver
Akinseloyin O, Jiang X, Palade V. A question-answering framework for automated abstract screening using large language models. Journal of the American Medical Informatics Association. 2024;31(9):1939-1952. doi:10.1093/jamia/ocae166
BibTeX
@article{Akinseloyin2024,
author = {Akinseloyin, Opeoluwa and Jiang, Xiaorui and Palade, Vasile},
title = {A question-answering framework for automated abstract screening using large language models},
journal = {Journal of the American Medical Informatics Association},
year = {2024},
volume = {31},
number = {9},
pages = {1939--1952},
doi = {10.1093/jamia/ocae166}
}
EUVIMED