EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Compact large language models for title and abstract screening in systematic reviews: An assessment of feasibility, accuracy, and workload reduction

Antonio Sciurti iD, Giuseppe Migliara, Leonardo Maria Siena, Claudia Isonne, Maria Roberta De Blasiis, Alessandra Sinopoli, Jessica Iera, Carolina Marzuillo, Corrado De Vito, Paolo Villari, Valentina Baccolini

Research Synthesis Methods · 2026-03 · 17 (2) · 332-347

DOI: 10.1017/rsm.2025.10044 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Research Synthesis Methods, online erschienen am 13.11.2025. Verfasst von Antonio Sciurti und 10 weiteren Autor:innen. Beteiligte Einrichtung: Department of Public Health and Infectious Diseases, University of Rome La Sapienza, Italy. Hauptthemen laut MEDLINE: Large Language Models, Review Literature as Topic. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 62 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Systematic reviews play a critical role in evidence-based research but are labor-intensive, especially during title and abstract screening. Compact large language models (LLMs) offer potential to automate this process, balancing time/cost requirements and accuracy. The aim of this study is to assess the feasibility, accuracy, and workload reduction by three compact LLMs (GPT-4o mini, Llama 3.1 8B, and Gemma 2 9B) in screening titles and abstracts. Records were sourced from three previously published systematic reviews and LLMs were requested to rate each record from 0 to 100 for inclusion, using a structured prompt. Predefined 25-, 50-, 75-rating thresholds were used to compute performance metrics (balanced accuracy, sensitivity, specificity, positive and negative predictive value, and workload-saving). Processing time and costs were registered. Across the systematic reviews, LLMs achieved high sensitivity (up to 100%) and low precision (below 10%) for records included by full text. Specificity and workload savings improved at higher thresholds, with the 50- and 75-rating thresholds offering optimal trade-offs. GPT-4o-mini, accessed via application programming interface, was the fastest model (~40 minutes max.) and had usage costs ($0.14–$1.93 per review). Llama 3.1-8B and Gemma 2-9B were run locally in longer times (~4 hours max.) and were free to use. LLMs were highly sensitive tools for the title/abstract screening process. High specificity values were reached, allowing for significant workload savings, at reasonable costs and processing time. Conversely, we found them to be imprecise. However, high sensitivity and workload reduction are key factors for their usage in the title/abstract screening phase of systematic reviews.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsReview Literature as TopicWorkloadSensitivity and SpecificityHumansReproducibility of ResultsFeasibility StudiesSoftwareAlgorithms

Stichwörter der Autor:innen

GPT-4o miniGemma 2 9BLlama 3.1 8Bartificial intelligencelarge language modelstitle and abstract screening

Autor:innen und Institutionen

  • Department of Public Health and Infectious Diseases, University of Rome La Sapienza, Italy.
  • Department of Life Sciences, Health, and Health Professions, Link Campus University, Italy.
  • Department of Prevention, Local Health Authority Rome 1, Italy.
  • Department of Infectious Diseases, Istituto Superiore di Sanità, Italy.

ORCID-Profile: Antonio Sciurti

Zitieren

APA

Sciurti, A., Migliara, G., Siena, L. M., Isonne, C., De Blasiis, M. R., Sinopoli, A., Iera, J., Marzuillo, C., De Vito, C., Villari, P., & Baccolini, V. (2026). Compact large language models for title and abstract screening in systematic reviews: An assessment of feasibility, accuracy, and workload reduction. Research Synthesis Methods, 17(2), 332-347. https://doi.org/10.1017/rsm.2025.10044

Vancouver

Sciurti A, Migliara G, Siena LM, Isonne C, De Blasiis MR, Sinopoli A, et al. Compact large language models for title and abstract screening in systematic reviews: An assessment of feasibility, accuracy, and workload reduction. Research Synthesis Methods. 2026;17(2):332-347. doi:10.1017/rsm.2025.10044

BibTeX
@article{Sciurti2026,
  author = {Sciurti, Antonio and Migliara, Giuseppe and Siena, Leonardo Maria and Isonne, Claudia and De Blasiis, Maria Roberta and Sinopoli, Alessandra and Iera, Jessica and Marzuillo, Carolina and De Vito, Corrado and Villari, Paolo and Baccolini, Valentina},
  title = {Compact large language models for title and abstract screening in systematic reviews: An assessment of feasibility, accuracy, and workload reduction},
  journal = {Research Synthesis Methods},
  year = {2026},
  volume = {17},
  number = {2},
  pages = {332--347},
  doi = {10.1017/rsm.2025.10044}
}
Kostenlos starten →