What level of automation is “good enough”? A benchmark of large language models for meta-analysis data extraction
Research Synthesis Methods · 2026-07 · 17 (4) · 671-692
DOI: 10.1017/rsm.2025.10066 ↗Kurz gefasst
Metaanalyse in der Zeitschrift Research Synthesis Methods, online erschienen am 26.01.2026. Verfasst von Lingbo Li und 2 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 57 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Automating data extraction from full-text randomized controlled trials for meta-analysis remains a significant challenge. This study evaluates the practical performance of three large language models (LLMs) (Gemini-2.0-flash, Grok-3, and GPT-4o-mini) across tasks involving statistical results, risk-of-bias assessments, and study-level characteristics in three medical domains: hypertension, diabetes, and orthopaedics. We tested four distinct prompting strategies (basic prompting, self-reflective prompting, model ensemble, and customized prompts) to determine how to improve extraction quality. All models demonstrate high precision but consistently suffer from poor recall by omitting key information. We found that customized prompts were the most effective, boosting recall by up to 15%. Based on this analysis, we propose a three-tiered set of guidelines for using LLMs in data extraction, matching data types to appropriate levels of automation based on task complexity and risk. Our study offers practical advice for automating data extraction in real-world meta-analyses, balancing LLM efficiency with expert oversight through targeted, task-specific automation.
Zitieren
APA
Li, L., Mathrani, A., & Susnjak, T. (2026). What level of automation is “good enough”? A benchmark of large language models for meta-analysis data extraction. Research Synthesis Methods, 17(4), 671-692. https://doi.org/10.1017/rsm.2025.10066
Vancouver
Li L, Mathrani A, Susnjak T. What level of automation is “good enough”? A benchmark of large language models for meta-analysis data extraction. Research Synthesis Methods. 2026;17(4):671-692. doi:10.1017/rsm.2025.10066
BibTeX
@article{Li2026,
author = {Li, Lingbo and Mathrani, Anuradha and Susnjak, Teo},
title = {What level of automation is “good enough”? A benchmark of large language models for meta-analysis data extraction},
journal = {Research Synthesis Methods},
year = {2026},
volume = {17},
number = {4},
pages = {671--692},
doi = {10.1017/rsm.2025.10066}
}
EUVIMED