EUVIMEDEUVIMED entdecken
Zurückgezogene PublikationSystematische ÜbersichtOpen Access · CC BY 4.0

RETRACTED: Human‐in‐the‐Loop Artificial Intelligence System for Systematic Literature Review: Methods and Validations for the AutoLit Review Software

Kevin M Kallmes iD, Jade Thurnham, Marius Sauca, Ranita Tarchand, Keith R Kallmes, Karl J Holub

Cochrane Evidence Synthesis and Methods · 2025-11 · 3 (6)

DOI: 10.1002/cesm.70059 ↗

Kurz gefasst

Zurückgezogene Publikation in der Zeitschrift Cochrane Evidence Synthesis and Methods, online erschienen am 25.10.2025. Verfasst von Kevin M Kallmes und 5 weiteren Autor:innen. Beteiligte Einrichtung: Nested Knowledge St. Paul Minnesota USA. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 41 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Introduction

While artificial intelligence (AI) tools have been utilized for individual stages within the systematic literature review (SLR) process, no tool has previously been shown to support each critical SLR step. In addition, the need for expert oversight has been recognized to ensure the quality of SLR findings. Here, we describe a complete methodology for utilizing our AI SLR tool with human‐in‐the‐loop curation workflows, as well as AI validations, time savings, and approaches to ensure compliance with best review practices.

Methods

SLRs require completing Search, Screening, and Extraction from relevant studies, with meta‐analysis and critical appraisal as relevant. We present a full methodological framework for completing SLRs utilizing our AutoLit software (Nested Knowledge). This system integrates AI models into the central steps in SLR: Search strategy generation, Dual Screening of Titles/Abstracts and Full Texts, and Extraction of qualitative and quantitative evidence. The system also offers manual Critical Appraisal and Insight drafting and fully‐automated Network Meta‐analysis. Validations comparing AI performance to experts are reported, and where relevant, time savings and ‘rapid review’ alternatives to the SLR workflow.

Results

Search strategy generation with the Smart Search AI can turn a Research Question into full Boolean strings with 76.8% and 79.6% Recall in two validation sets. Supervised machine learning tools can achieve 82–97% Recall in reviewer‐level Screening. Population, Interventions/Comparators, and Outcomes (PICOs) extraction achieved F1 of 0.74; accuracy for study type, location, and size were 74%, 78%, and 91%, respectively. Time savings of 50% in Abstract Screening and 70–80% in qualitative extraction were reported. Extraction of user‐specified qualitative and quantitative tags and data elements remains exploratory and requires human curation for SLRs.

Conclusion

AI systems can support high‐quality, human‐in‐the‐loop execution of key SLR stages. Transparency, replicability, and expert oversight are central to the use of AI SLR tools.

Schlagwörter

Stichwörter der Autor:innen

artificial intelligenceevidence synthesishuman‐in‐the‐loopmeta‐analysissystematic literature review

Autor:innen und Institutionen

  • Nested Knowledge St. Paul Minnesota USA.

ORCID-Profile: Kevin M Kallmes

Zitieren

APA

Kallmes, K. M., Thurnham, J., Sauca, M., Tarchand, R., Kallmes, K. R., & Holub, K. J. (2025). RETRACTED: Human‐in‐the‐Loop Artificial Intelligence System for Systematic Literature Review: Methods and Validations for the AutoLit Review Software. Cochrane Evidence Synthesis and Methods, 3(6). https://doi.org/10.1002/cesm.70059

Vancouver

Kallmes KM, Thurnham J, Sauca M, Tarchand R, Kallmes KR, Holub KJ. RETRACTED: Human‐in‐the‐Loop Artificial Intelligence System for Systematic Literature Review: Methods and Validations for the AutoLit Review Software. Cochrane Evidence Synthesis and Methods. 2025;3(6). doi:10.1002/cesm.70059

BibTeX
@article{Kallmes2025,
  author = {Kallmes, Kevin M and Thurnham, Jade and Sauca, Marius and Tarchand, Ranita and Kallmes, Keith R and Holub, Karl J},
  title = {RETRACTED: Human‐in‐the‐Loop Artificial Intelligence System for Systematic Literature Review: Methods and Validations for the AutoLit Review Software},
  journal = {Cochrane Evidence Synthesis and Methods},
  year = {2025},
  volume = {3},
  number = {6},
  doi = {10.1002/cesm.70059}
}
Kostenlos starten →