Using Artificial Intelligence Tools as Second Reviewers for Data Extraction in Systematic Reviews: A Performance Comparison of Two AI Tools Against Human Reviewers
Cochrane Evidence Synthesis and Methods · 2025-07 · 3 (4)
DOI: 10.1002/cesm.70036 ↗Kurz gefasst
Fachartikel in der Zeitschrift Cochrane Evidence Synthesis and Methods, online erschienen am 14.07.2025. Verfasst von T Helms Andersen und 4 weiteren Autor:innen. Beteiligte Einrichtung: Copenhagen University Hospital-Steno Diabetes Center Copenhagen Herlev Denmark. Die Arbeit wurde bisher 38-mal zitiert (Crossref) und nennt 33 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Background
Systematic reviews are essential but time‐consuming and expensive. Large language models (LLMs) and artificial intelligence (AI) tools could potentially automate data extraction, but no comprehensive workflow has been tested for different review types.
Objective
To evaluate Elicit's and ChatGPT's abilities to extract data from journal articles as a replacement for one of two human data extractors in systematic reviews.
Methods
Human‐extracted data from three systematic reviews (30 articles in total) was compared to data extracted by Elicit and ChatGPT. The AI tools extracted population characteristics, study design, and review‐specific variables. Performance metrics were calculated against human double‐extracted data as the gold standard, followed by a detailed error analysis.
Results
Precision, recall and F1‐score were all 92% for Elicit and 91%, 89% and 90% for ChatGPT. Recall was highest for study design (Elicit: 100%; ChatGPT: 90%) and population characteristics (Elicit: 100%; ChatGPT: 97%), while review‐specific variables achieved 77% in Elicit and 80% in ChatGPT. Elicit had four instances of confabulation while ChatGPT had three. There was no significant difference between the two AI tools' performance (recall difference: 3.3% points, 95% CI: –5.2%–11.9%, p = 0.445).
Conclusion
AI tools demonstrated high and similar performance in data extraction compared to human reviewers, particularly for standardized variables. Error analysis revealed confabulations in 4% of data points. We propose adopting AI‐assisted extraction to replace the second human extractor, with the second human instead focusing on reconciling discrepancies between AI and the primary human extractor.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Copenhagen University Hospital-Steno Diabetes Center Copenhagen Herlev Denmark.
ORCID-Profile: T Helms Andersen, T M Marcussen, A D Termannsen, T W H Lawaetz, O Nørgaard
Zitieren
APA
Helms Andersen, T., Marcussen, T. M., Termannsen, A. D., Lawaetz, T. W. H., & Nørgaard, O. (2025). Using Artificial Intelligence Tools as Second Reviewers for Data Extraction in Systematic Reviews: A Performance Comparison of Two AI Tools Against Human Reviewers. Cochrane Evidence Synthesis and Methods, 3(4). https://doi.org/10.1002/cesm.70036
Vancouver
Helms Andersen T, Marcussen TM, Termannsen AD, Lawaetz TWH, Nørgaard O. Using Artificial Intelligence Tools as Second Reviewers for Data Extraction in Systematic Reviews: A Performance Comparison of Two AI Tools Against Human Reviewers. Cochrane Evidence Synthesis and Methods. 2025;3(4). doi:10.1002/cesm.70036
BibTeX
@article{HelmsAndersen2025,
author = {Helms Andersen, T and Marcussen, T M and Termannsen, A D and Lawaetz, T W H and Nørgaard, O},
title = {Using Artificial Intelligence Tools as Second Reviewers for Data Extraction in Systematic Reviews: A Performance Comparison of Two AI Tools Against Human Reviewers},
journal = {Cochrane Evidence Synthesis and Methods},
year = {2025},
volume = {3},
number = {4},
doi = {10.1002/cesm.70036}
}
EUVIMED