EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

A question-answering framework for automated abstract screening using large language models

Opeoluwa Akinseloyin, Xiaorui Jiang iD, Vasile Palade iD

Journal of the American Medical Informatics Association · 01.09.2024 · 31 (9) · 1939-1952

DOI: 10.1093/jamia/ocae166 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 23.07.2024. Verfasst von Opeoluwa Akinseloyin und 2 weiteren Autor:innen. Beteiligte Einrichtung: Centre for Computational Science and Mathematical Modelling, Coventry University, Coventry CV1 2TT, United Kingdom. Hauptthemen laut MEDLINE: Natural Language Processing, Artificial Intelligence. Die Arbeit wurde bisher 36-mal zitiert (Crossref) und nennt 73 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: National Planning Office of Philosophy and Social Science of China, International Exchange Scheme of the Royal Society of the United Kingdom, Research Excellence Development Framework award of Coventry University.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

This paper aims to address the challenges in abstract screening within systematic reviews (SR) by leveraging the zero-shot capabilities of large language models (LLMs).

Methods

We employ LLM to prioritize candidate studies by aligning abstracts with the selection criteria outlined in an SR protocol. Abstract screening was transformed into a novel question-answering (QA) framework, treating each selection criterion as a question addressed by LLM. The framework involves breaking down the selection criteria into multiple questions, properly prompting LLM to answer each question, scoring and re-ranking each answer, and combining the responses to make nuanced inclusion or exclusion decisions. Results and Discussion Large-scale validation was performed on the benchmark of CLEF eHealth 2019 Task 2: Technology-Assisted Reviews in Empirical Medicine. Focusing on GPT-3.5 as a case study, the proposed QA framework consistently exhibited a clear advantage over traditional information retrieval approaches and bespoke BERT-family models that were fine-tuned for prioritizing candidate studies (ie, from the BERT to PubMedBERT) across 31 datasets of 4 categories of SRs, underscoring their high potential in facilitating abstract screening. The experiments also showcased the viability of using selection criteria as a query for reference prioritization. The experiments also showcased the viability of the framework using different LLMs.

Conclusion

Investigation justified the indispensable value of leveraging selection criteria to improve the performance of automated abstract screening. LLMs demonstrated proficiency in prioritizing candidate studies for abstract screening using the proposed QA framework. Significant performance improvements were obtained by re-ranking answers using the semantic alignment between abstracts and selection criteria. This further highlighted the pertinence of utilizing selection criteria to enhance abstract screening.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language ProcessingArtificial IntelligenceHumansAbstracting and IndexingInformation Storage and RetrievalSystematic Reviews as Topic

Stichwörter der Autor:innen

abstract screeningautomated systematic reviewlarge language modelquestion answeringzero-shot re-ranking

Autor:innen und Institutionen

  • Centre for Computational Science and Mathematical Modelling, Coventry University, Coventry CV1 2TT, United Kingdom.
  • Information School, The University of Sheffield, Sheffield S10 2AH, United Kingdom.

ORCID-Profile: Xiaorui Jiang, Vasile Palade

Förderung

  • National Planning Office of Philosophy and Social Science of China
  • International Exchange Scheme of the Royal Society of the United Kingdom
  • Research Excellence Development Framework award of Coventry University
  • Coventry University
  • International Exchange Scheme
  • Royal Society of the United Kingdom

Zitieren

APA

Akinseloyin, O., Jiang, X., & Palade, V. (2024). A question-answering framework for automated abstract screening using large language models. Journal of the American Medical Informatics Association, 31(9), 1939-1952. https://doi.org/10.1093/jamia/ocae166

Vancouver

Akinseloyin O, Jiang X, Palade V. A question-answering framework for automated abstract screening using large language models. Journal of the American Medical Informatics Association. 2024;31(9):1939-1952. doi:10.1093/jamia/ocae166

BibTeX
@article{Akinseloyin2024,
  author = {Akinseloyin, Opeoluwa and Jiang, Xiaorui and Palade, Vasile},
  title = {A question-answering framework for automated abstract screening using large language models},
  journal = {Journal of the American Medical Informatics Association},
  year = {2024},
  volume = {31},
  number = {9},
  pages = {1939--1952},
  doi = {10.1093/jamia/ocae166}
}
Kostenlos starten →