EUVIMEDEUVIMED entdecken
VergleichsstudieMetaanalyseOpen Access · CC BY-ND 4.0

Prompt engineering of large language models for paper screening in medical meta-analyses and systematic reviews: A prospective comparative study

Till J Adam iD, Salma A S Abosabie, Max Dittmer, Elise Wolf iD, Sara A Abosabie, Clara Behnke, Felix Baier, Annabelle Weickmann, Ludwig Köser, Christoph U Correll, Niklas Rutsch

Research Synthesis Methods · 2026-09 · 17 (5) · 939-956

DOI: 10.1017/rsm.2026.10093 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift Research Synthesis Methods, online erschienen am 14.04.2026. Verfasst von Till J Adam und 10 weiteren Autor:innen. Beteiligte Einrichtung: Department of Child and Adolescent Psychiatry, Psychosomatic Medicine and Psychotherapy, Charité - Universitätsmedizin Berlin, Germany. Hauptthemen laut MEDLINE: Large Language Models, Meta-Analysis as Topic, Systematic Reviews as Topic. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 49 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-ND 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Interest in large language models (LLMs) as a tool for meta-analyses and systematic reviews (MA/SRs) is growing. We prospectively developed 515 unique prompts by predefined screening-related categories and tested with open-access LLMs (Llama, Mistral) against four gold-standard MA/SRs from different medical fields published after the LLMs’ training cut-offs, using a Python-based pipeline. Heterogeneity between prompts was quantified, and hypothetical workload/cost reduction with top-performing prompts calculated. Across 12,360 pipeline runs, LLMs versus MA/SRs reached average recall/sensitivity = 83.6 ± 17.0%, precision = 18.5 ± 15.6%, specificity = 36.6 ± 23.7% F1-score = 27.6 ± 17.2%, and accuracy = 61.1 ± 11.0%. F1-scores were significantly higher when prompts focused on methods (0.78 ± 0.40%), explicitly mentioned MA/SR screening (0.81 ± 0.37%), included the comparison MA/SR’s title (5.64 ± 0.37%) or selection criteria (8.05 ± 0.68%), and with more LLM parameters (70b = 4.48 ± 0.31%, 123b = 7.77 ± 0.31%), but lower when screening abstracts instead of titles (−3.67 ± 0.28%). In LLM-base preselection, top-performing F1-score prompts (recall/sensitivity = 72.2%, specificity = 66.1%, precision = 28.6%) would reduce screening demands by 34.5%−37.5%, saving 8.4–8.8 weeks of work and 17,592–18,552. Recall/sensitivity increased with less MA/SR information contrasting F1-score results, which highlights a recall/sensitivity-precision/specificity trade-off. F1-score increased with detailed MA/SR information, while recall/sensitivity increased with shorter, zeroshot prompts. We provide the first prospectively assessed prompt engineering framework for early-stage LLM-based paper screening across medical fields. The publicly available Python pipeline and full prompt list used here support further development of LLM-based evidence synthesis.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsMeta-Analysis as TopicSystematic Reviews as TopicHumansProspective StudiesSoftwarePaperSensitivity and SpecificityAlgorithmsReproducibility of ResultsReview Literature as Topic

Stichwörter der Autor:innen

evidence-based medicinelarge language modelsmeta-analysis as topicprompt engineeringsystematic reviews as topic

Autor:innen und Institutionen

  • Department of Child and Adolescent Psychiatry, Psychosomatic Medicine and Psychotherapy, Charité - Universitätsmedizin Berlin, Germany.
  • Department of Radiology, Charite - Universitatsmedizin Berlin, Germany.
  • Department of Physics, Technische Universität Berlin, Germany.
  • School of Business Informatics and Mathematics, Universität Mannheim, Germany.
  • Department of Psychology, Humboldt-Universitat zu Berlin, Germany.
  • Department of Psychology, University of Potsdam, Germany.

ORCID-Profile: Till J Adam, Elise Wolf

Zitieren

APA

Adam, T. J., Abosabie, S. A. S., Dittmer, M., Wolf, E., Abosabie, S. A., Behnke, C., Baier, F., Weickmann, A., Köser, L., Correll, C. U., & Rutsch, N. (2026). Prompt engineering of large language models for paper screening in medical meta-analyses and systematic reviews: A prospective comparative study. Research Synthesis Methods, 17(5), 939-956. https://doi.org/10.1017/rsm.2026.10093

Vancouver

Adam TJ, Abosabie SAS, Dittmer M, Wolf E, Abosabie SA, Behnke C, et al. Prompt engineering of large language models for paper screening in medical meta-analyses and systematic reviews: A prospective comparative study. Research Synthesis Methods. 2026;17(5):939-956. doi:10.1017/rsm.2026.10093

BibTeX
@article{Adam2026,
  author = {Adam, Till J and Abosabie, Salma A S and Dittmer, Max and Wolf, Elise and Abosabie, Sara A and Behnke, Clara and Baier, Felix and Weickmann, Annabelle and Köser, Ludwig and Correll, Christoph U and Rutsch, Niklas},
  title = {Prompt engineering of large language models for paper screening in medical meta-analyses and systematic reviews: A prospective comparative study},
  journal = {Research Synthesis Methods},
  year = {2026},
  volume = {17},
  number = {5},
  pages = {939--956},
  doi = {10.1017/rsm.2026.10093}
}
Kostenlos starten →