EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Evaluating the AI Tool “Elicit” as a Semi-Automated Second Reviewer for Data Extraction in Systematic Reviews: A Proof-of-Concept

Frederic Hilkenmeier, Marie Pelzer, Christian Stierle, Jakob Fink-Lamotte

Social Science Computer Review · 03.12.2025

DOI: 10.1177/08944393251404052 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Social Science Computer Review, online erschienen am 03.12.2025. Verfasst von Frederic Hilkenmeier und 3 weiteren Autor:innen. Die Arbeit wurde bisher 17-mal zitiert (Crossref) und nennt 79 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Systematic reviews are essential for evidence synthesis but often require extensive time and resources, especially during data extraction. This proof-of-concept study evaluates the performance of Elicit , an AI tool specifically developed to support systematic reviews, in the context of a systematic review on psychological factors in dermatological conditions. We compared Elicit’s automated data extraction with manually extracted data across 43 studies and 602 data points. Both were assessed against a consensus-based ground truth. Elicit achieved an overall accuracy of 81.4%, compared to 86.7% for human reviewers—a difference that was not statistically significant. In cases where Elicit and the human reviewer extracted the same information, this information was correct in 100% of instances, suggesting that agreement between human and machine may serve as a reliable proxy for validity. Based on these results, we propose a semi-automated workflow in which Elicit functions as a second reviewer, reducing workload while maintaining high data quality. Our results demonstrate that domain-specific AI tools can effectively augment data extraction in systematic reviews, especially in settings with limited time or personnel.

Zitieren

APA

Hilkenmeier, F., Pelzer, M., Stierle, C., & Fink-Lamotte, J. (2025). Evaluating the AI Tool “Elicit” as a Semi-Automated Second Reviewer for Data Extraction in Systematic Reviews: A Proof-of-Concept. Social Science Computer Review. https://doi.org/10.1177/08944393251404052

Vancouver

Hilkenmeier F, Pelzer M, Stierle C, Fink-Lamotte J. Evaluating the AI Tool “Elicit” as a Semi-Automated Second Reviewer for Data Extraction in Systematic Reviews: A Proof-of-Concept. Social Science Computer Review. 2025. doi:10.1177/08944393251404052

BibTeX
@article{Hilkenmeier2025,
  author = {Hilkenmeier, Frederic and Pelzer, Marie and Stierle, Christian and Fink-Lamotte, Jakob},
  title = {Evaluating the AI Tool “Elicit” as a Semi-Automated Second Reviewer for Data Extraction in Systematic Reviews: A Proof-of-Concept},
  journal = {Social Science Computer Review},
  year = {2025},
  doi = {10.1177/08944393251404052}
}
Kostenlos starten →