EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Using Artificial Intelligence Tools as Second Reviewers for Data Extraction in Systematic Reviews: A Performance Comparison of Two AI Tools Against Human Reviewers

T Helms Andersen iD, T M Marcussen iD, A D Termannsen iD, T W H Lawaetz iD, O Nørgaard iD

Cochrane Evidence Synthesis and Methods · 2025-07 · 3 (4)

DOI: 10.1002/cesm.70036 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Cochrane Evidence Synthesis and Methods, online erschienen am 14.07.2025. Verfasst von T Helms Andersen und 4 weiteren Autor:innen. Beteiligte Einrichtung: Copenhagen University Hospital-Steno Diabetes Center Copenhagen Herlev Denmark. Die Arbeit wurde bisher 38-mal zitiert (Crossref) und nennt 33 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Systematic reviews are essential but time‐consuming and expensive. Large language models (LLMs) and artificial intelligence (AI) tools could potentially automate data extraction, but no comprehensive workflow has been tested for different review types.

Objective

To evaluate Elicit's and ChatGPT's abilities to extract data from journal articles as a replacement for one of two human data extractors in systematic reviews.

Methods

Human‐extracted data from three systematic reviews (30 articles in total) was compared to data extracted by Elicit and ChatGPT. The AI tools extracted population characteristics, study design, and review‐specific variables. Performance metrics were calculated against human double‐extracted data as the gold standard, followed by a detailed error analysis.

Results

Precision, recall and F1‐score were all 92% for Elicit and 91%, 89% and 90% for ChatGPT. Recall was highest for study design (Elicit: 100%; ChatGPT: 90%) and population characteristics (Elicit: 100%; ChatGPT: 97%), while review‐specific variables achieved 77% in Elicit and 80% in ChatGPT. Elicit had four instances of confabulation while ChatGPT had three. There was no significant difference between the two AI tools' performance (recall difference: 3.3% points, 95% CI: –5.2%–11.9%, p = 0.445).

Conclusion

AI tools demonstrated high and similar performance in data extraction compared to human reviewers, particularly for standardized variables. Error analysis revealed confabulations in 4% of data points. We propose adopting AI‐assisted extraction to replace the second human extractor, with the second human instead focusing on reconciling discrepancies between AI and the primary human extractor.

Schlagwörter

Stichwörter der Autor:innen

ChatGPTElicitartificial intelligencedata extractionlarge language modelsresearch methodologysystematic review methodology

Autor:innen und Institutionen

  • Copenhagen University Hospital-Steno Diabetes Center Copenhagen Herlev Denmark.

ORCID-Profile: T Helms Andersen, T M Marcussen, A D Termannsen, T W H Lawaetz, O Nørgaard

Zitieren

APA

Helms Andersen, T., Marcussen, T. M., Termannsen, A. D., Lawaetz, T. W. H., & Nørgaard, O. (2025). Using Artificial Intelligence Tools as Second Reviewers for Data Extraction in Systematic Reviews: A Performance Comparison of Two AI Tools Against Human Reviewers. Cochrane Evidence Synthesis and Methods, 3(4). https://doi.org/10.1002/cesm.70036

Vancouver

Helms Andersen T, Marcussen TM, Termannsen AD, Lawaetz TWH, Nørgaard O. Using Artificial Intelligence Tools as Second Reviewers for Data Extraction in Systematic Reviews: A Performance Comparison of Two AI Tools Against Human Reviewers. Cochrane Evidence Synthesis and Methods. 2025;3(4). doi:10.1002/cesm.70036

BibTeX
@article{HelmsAndersen2025,
  author = {Helms Andersen, T and Marcussen, T M and Termannsen, A D and Lawaetz, T W H and Nørgaard, O},
  title = {Using Artificial Intelligence Tools as Second Reviewers for Data Extraction in Systematic Reviews: A Performance Comparison of Two AI Tools Against Human Reviewers},
  journal = {Cochrane Evidence Synthesis and Methods},
  year = {2025},
  volume = {3},
  number = {4},
  doi = {10.1002/cesm.70036}
}
Kostenlos starten →