Prompt engineering of large language models for paper screening in medical meta-analyses and systematic reviews: A prospective comparative study
Research Synthesis Methods · 2026-09 · 17 (5) · 939-956
DOI: 10.1017/rsm.2026.10093 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Research Synthesis Methods, online erschienen am 14.04.2026. Verfasst von Till J Adam und 10 weiteren Autor:innen. Beteiligte Einrichtung: Department of Child and Adolescent Psychiatry, Psychosomatic Medicine and Psychotherapy, Charité - Universitätsmedizin Berlin, Germany. Hauptthemen laut MEDLINE: Large Language Models, Meta-Analysis as Topic, Systematic Reviews as Topic. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 49 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-ND 4.0).
Abstract (Originalsprache)
Interest in large language models (LLMs) as a tool for meta-analyses and systematic reviews (MA/SRs) is growing. We prospectively developed 515 unique prompts by predefined screening-related categories and tested with open-access LLMs (Llama, Mistral) against four gold-standard MA/SRs from different medical fields published after the LLMs’ training cut-offs, using a Python-based pipeline. Heterogeneity between prompts was quantified, and hypothetical workload/cost reduction with top-performing prompts calculated. Across 12,360 pipeline runs, LLMs versus MA/SRs reached average recall/sensitivity = 83.6 ± 17.0%, precision = 18.5 ± 15.6%, specificity = 36.6 ± 23.7% F1-score = 27.6 ± 17.2%, and accuracy = 61.1 ± 11.0%. F1-scores were significantly higher when prompts focused on methods (0.78 ± 0.40%), explicitly mentioned MA/SR screening (0.81 ± 0.37%), included the comparison MA/SR’s title (5.64 ± 0.37%) or selection criteria (8.05 ± 0.68%), and with more LLM parameters (70b = 4.48 ± 0.31%, 123b = 7.77 ± 0.31%), but lower when screening abstracts instead of titles (−3.67 ± 0.28%). In LLM-base preselection, top-performing F1-score prompts (recall/sensitivity = 72.2%, specificity = 66.1%, precision = 28.6%) would reduce screening demands by 34.5%−37.5%, saving 8.4–8.8 weeks of work and 17,592–18,552. Recall/sensitivity increased with less MA/SR information contrasting F1-score results, which highlights a recall/sensitivity-precision/specificity trade-off. F1-score increased with detailed MA/SR information, while recall/sensitivity increased with shorter, zeroshot prompts. We provide the first prospectively assessed prompt engineering framework for early-stage LLM-based paper screening across medical fields. The publicly available Python pipeline and full prompt list used here support further development of LLM-based evidence synthesis.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Child and Adolescent Psychiatry, Psychosomatic Medicine and Psychotherapy, Charité - Universitätsmedizin Berlin, Germany.
- Department of Radiology, Charite - Universitatsmedizin Berlin, Germany.
- Department of Physics, Technische Universität Berlin, Germany.
- School of Business Informatics and Mathematics, Universität Mannheim, Germany.
- Department of Psychology, Humboldt-Universitat zu Berlin, Germany.
- Department of Psychology, University of Potsdam, Germany.
ORCID-Profile: Till J Adam, Elise Wolf
Zitieren
APA
Adam, T. J., Abosabie, S. A. S., Dittmer, M., Wolf, E., Abosabie, S. A., Behnke, C., Baier, F., Weickmann, A., Köser, L., Correll, C. U., & Rutsch, N. (2026). Prompt engineering of large language models for paper screening in medical meta-analyses and systematic reviews: A prospective comparative study. Research Synthesis Methods, 17(5), 939-956. https://doi.org/10.1017/rsm.2026.10093
Vancouver
Adam TJ, Abosabie SAS, Dittmer M, Wolf E, Abosabie SA, Behnke C, et al. Prompt engineering of large language models for paper screening in medical meta-analyses and systematic reviews: A prospective comparative study. Research Synthesis Methods. 2026;17(5):939-956. doi:10.1017/rsm.2026.10093
BibTeX
@article{Adam2026,
author = {Adam, Till J and Abosabie, Salma A S and Dittmer, Max and Wolf, Elise and Abosabie, Sara A and Behnke, Clara and Baier, Felix and Weickmann, Annabelle and Köser, Ludwig and Correll, Christoph U and Rutsch, Niklas},
title = {Prompt engineering of large language models for paper screening in medical meta-analyses and systematic reviews: A prospective comparative study},
journal = {Research Synthesis Methods},
year = {2026},
volume = {17},
number = {5},
pages = {939--956},
doi = {10.1017/rsm.2026.10093}
}
EUVIMED