EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Optimal large language models to screen citations for systematic reviews

Takehiko Oami, Yohei Okada, Taka-Aki Nakada

Research Synthesis Methods · 2025-11 · 16 (6) · 859-875

DOI: 10.1017/rsm.2025.10014 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Research Synthesis Methods, online erschienen am 23.06.2025. Verfasst von Takehiko Oami und 2 weiteren Autor:innen. Beteiligte Einrichtung: Department of Emergency and Critical Care Medicine, Chiba University Graduate School of Medicine, Chiba, Japan. Hauptthemen laut MEDLINE: Large Language Models, Systematic Reviews as Topic. Die Arbeit wurde bisher 22-mal zitiert (Crossref) und nennt 48 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recent studies highlight the potential of large language models (LLMs) in citation screening for systematic reviews; however, the efficiency of individual LLMs for this application remains unclear. This study aimed to compare accuracy, time-related efficiency, cost, and consistency across four LLMs—GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet, and Llama 3.3 70B—for literature screening tasks. The models screened for clinical questions from the Japanese Clinical Practice Guidelines for the Management of Sepsis and Septic Shock 2024. Sensitivity and specificity were calculated for each model based on conventional citation screening results for qualitative assessment. We also recorded the time and cost of screening and assessed consistency to verify reproducibility. A post hoc analysis explored whether integrating outputs from multiple models could enhance screening accuracy. GPT-4o and Llama 3.3 70B achieved high specificity but lower sensitivity, while Gemini 1.5 Pro and Claude 3.5 Sonnet exhibited higher sensitivity at the cost of lower specificity. Citation screening times and costs varied, with GPT-4o being the fastest and Llama 3.3 70B the most cost-effective. Consistency was comparable among the models. An ensemble approach combining model outputs improved sensitivity but increased the number of false positives, requiring additional review effort. Each model demonstrated distinct strengths, effectively streamlining citation screening by saving time and reducing workload. However, reviewing false positives remains a challenge. Combining models may enhance sensitivity, indicating the potential of LLMs to optimize systematic review workflows.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsSystematic Reviews as TopicReproducibility of ResultsSensitivity and SpecificityHumansCost-Benefit AnalysisJapanAlgorithms

Stichwörter der Autor:innen

citation screeningclinical practice guidelinesgenerative AIlarge language modelssystematic review

Autor:innen und Institutionen

  • Department of Emergency and Critical Care Medicine, Chiba University Graduate School of Medicine, Chiba, Japan.
  • Department of Preventive Services, Kyoto University Graduate School of Medicine, Kyoto, Japan.
  • Pre-hospital and Emergency Research Centre, Health Services and Systems Research, Duke-NUS Medical School, https://ror.org/01tgyzw49National University of SingaporeSingapore.

Zitieren

APA

Oami, T., Okada, Y., & Nakada, T. A. (2025). Optimal large language models to screen citations for systematic reviews. Research Synthesis Methods, 16(6), 859-875. https://doi.org/10.1017/rsm.2025.10014

Vancouver

Oami T, Okada Y, Nakada TA. Optimal large language models to screen citations for systematic reviews. Research Synthesis Methods. 2025;16(6):859-875. doi:10.1017/rsm.2025.10014

BibTeX
@article{Oami2025,
  author = {Oami, Takehiko and Okada, Yohei and Nakada, Taka-Aki},
  title = {Optimal large language models to screen citations for systematic reviews},
  journal = {Research Synthesis Methods},
  year = {2025},
  volume = {16},
  number = {6},
  pages = {859--875},
  doi = {10.1017/rsm.2025.10014}
}
Kostenlos starten →