EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain

Fabio Dennstädt iD, Johannes Zink, Paul Martin Putora, Janna Hastings, Nikola Cihoric

Systematic Reviews · 15.06.2024 · 13 (1)

DOI: 10.1186/s13643-024-02575-4 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Systematic Reviews, online erschienen am 15.06.2024. Verfasst von Fabio Dennstädt und 4 weiteren Autor:innen. Beteiligte Einrichtung: Department of Radiation Oncology, Cantonal Hospital of St. Gallen, St. Gallen, Switzerland. fabiodennstaedt@gmx.de. Hauptthemen laut MEDLINE: Natural Language Processing. Die Arbeit wurde bisher 168-mal zitiert (Crossref) und nennt 75 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Systematically screening published literature to determine the relevant publications to synthesize in a review is a time-consuming and difficult task. Large language models (LLMs) are an emerging technology with promising capabilities for the automation of language-related tasks that may be useful for such a purpose.

Methods

LLMs were used as part of an automated system to evaluate the relevance of publications to a certain topic based on defined criteria and based on the title and abstract of each publication. A Python script was created to generate structured prompts consisting of text strings for instruction, title, abstract, and relevant criteria to be provided to an LLM. The relevance of a publication was evaluated by the LLM on a Likert scale (low relevance to high relevance). By specifying a threshold, different classifiers for inclusion/exclusion of publications could then be defined. The approach was used with four different openly available LLMs on ten published data sets of biomedical literature reviews and on a newly human-created data set for a hypothetical new systematic literature review.

Results

The performance of the classifiers varied depending on the LLM being used and on the data set analyzed. Regarding sensitivity/specificity, the classifiers yielded 94.48%/31.78% for the FlanT5 model, 97.58%/19.12% for the OpenHermes-NeuralChat model, 81.93%/75.19% for the Mixtral model and 97.58%/38.34% for the Platypus 2 model on the ten published data sets. The same classifiers yielded 100% sensitivity at a specificity of 12.58%, 4.54%, 62.47%, and 24.74% on the newly created data set. Changing the standard settings of the approach (minor adaption of instruction prompt and/or changing the range of the Likert scale from 1–5 to 1–10) had a considerable impact on the performance.

Conclusions

LLMs can be used to evaluate the relevance of scientific publications to a certain review topic and classifiers based on such an approach show some promising results. To date, little is known about how well such systems would perform if used prospectively when conducting systematic literature reviews and what further implications this might have. However, it is likely that in the future researchers will increasingly use LLMs for evaluating and classifying scientific publications.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language ProcessingBiomedical ResearchLanguageSystematic Reviews as Topic

Stichwörter der Autor:innen

BiomedicineLarge language modelsNatural language processingSystematic literature reviewTitle and abstract screening

Autor:innen und Institutionen

  • Department of Radiation Oncology, Cantonal Hospital of St. Gallen, St. Gallen, Switzerland. fabiodennstaedt@gmx.de.
  • Department of Radiation Oncology, Inselspital, Bern University Hospital and University of Bern, Bern, Switzerland. fabiodennstaedt@gmx.de.
  • Institute for Computer Science, University of Würzburg, Würzburg, Germany.
  • Department of Radiation Oncology, Cantonal Hospital of St. Gallen, St. Gallen, Switzerland.
  • Department of Radiation Oncology, Inselspital, Bern University Hospital and University of Bern, Bern, Switzerland.
  • Institute for Implementation Science in Health Care, University of Zurich, Zurich, Switzerland.

ORCID-Profile: Fabio Dennstädt

Zitieren

APA

Dennstädt, F., Zink, J., Putora, P. M., Hastings, J., & Cihoric, N. (2024). Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain. Systematic Reviews, 13(1). https://doi.org/10.1186/s13643-024-02575-4

Vancouver

Dennstädt F, Zink J, Putora PM, Hastings J, Cihoric N. Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain. Systematic Reviews. 2024;13(1). doi:10.1186/s13643-024-02575-4

BibTeX
@article{Dennstdt2024,
  author = {Dennstädt, Fabio and Zink, Johannes and Putora, Paul Martin and Hastings, Janna and Cihoric, Nikola},
  title = {Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain},
  journal = {Systematic Reviews},
  year = {2024},
  volume = {13},
  number = {1},
  doi = {10.1186/s13643-024-02575-4}
}
Kostenlos starten →