EUVIMEDAtrasti EUVIMED
Mixed-Methods-StudieStudienprotokollOpen access

Evaluating the Methodological Quality of Artificial Intelligence–Assisted Systematic Reviews: Protocol for a Mixed Methods Meta-Research Study

Mohammad Jay iD, Mary Morgan iD, Sharon Elizabeth Straus iD, Emma Wilson iD, Rinku Sutradhar iD, Catherine Yu iD, Lesley Gotlib Conn iD, Christoffer Dharma iD, Lorraine Lipscombe iD, Antoine Eskander iD

JMIR Research Protocols · 2026-05-14 · 15 · e90588-e90588

DOI: 10.2196/90588 ↗

At a glance

Mixed-Methods-Studie in JMIR Research Protocols, published online on 2026-05-14. Written by Mohammad Jay and 9 co-authors. Institution: Department of Medicine, Division of Endocrinology, University of Toronto, 2075 Bayview Avenue, Toronto, ON, M4N 3M5, Canada, 1 416-480-6705, 1 416-48…. Major MEDLINE topics: Systematic Reviews as Topic, Artificial Intelligence, Research Design. Cited 1 times so far (Crossref); 41 references. The full text is freely available in PubMed Central.

SantraukaSubjectsBibliografiniai duomenysCiteRelated studies

Abstract (original language)

Background

Artificial intelligence (AI), including large language models (LLMs), is increasingly integrated into systematic review (SR) workflows. AI tools may accelerate searching, screening, data extraction, and reporting, but their effects on methodological quality, reporting completeness, transparency, and reproducibility remain uncertain. Existing evaluations largely examine isolated tasks, and inconsistent disclosure of AI use limits reproducibility and oversight.

Objective

This 4-phase mixed methods meta-research study will (1) compare the methodological quality of AI-assisted versus traditional SRs; (2) refine, finalize, and apply a preliminary AI Transparency and Disclosure Index (AITDI); (3) evaluate reproducibility by comparing outputs across repeated runs of the same AI model, across different AI models, and between AI models and human reviewers at multiple SR stages; and (4) explore knowledge user perspectives on rigor, transparency, and trust in AI-assisted SRs.

Methods

We will conduct a matched cohort analysis of SRs published from 2023 to 2025 in biomedical journals. Each AI-assisted SR will be matched 1:2 with traditional SRs by publication year, clinical domain, review type, and meta-analysis status. Two independent reviewers will apply A Measurement Tool to Assess Systematic Reviews, version 2 (AMSTAR 2; methodological quality), PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 (reporting completeness), and, when applicable, Risk of Bias in SRs (ROBIS; risk-of-bias rigor). A preliminary AITDI will be refined and then applied to all AI-assisted SRs. Reproducibility will be assessed using SR-derived task sets to compare outputs across repeated runs of the same model, across different models, and between AI and human reviewers at key SR stages. Semistructured interviews with authors, editors, clinicians, policymakers, and patient partners will be analyzed using reflexive thematic analysis.

Results

As of December 2025, the study has been preregistered on the Open Science Framework (OSF; DOI: 10.17605/OSF.IO/Q5JRW), the search strategy has been finalized, and title/abstract screening has begun. Data extraction is planned for March-May 2026, followed by AITDI refinement and reproducibility testing from May 2026 to October 2026. Qualitative interviews are anticipated from October 2026 to February 2027, with final analyses by April 2027 and dissemination planned for mid-2027.

Conclusions

This study will provide one of the first empirical comparisons of methodological quality, transparency, and reproducibility of AI-assisted versus traditional SRs in the LLM era. Findings will inform expectations for responsible AI integration and support refinement of reporting and methodological best practices, including future development of AI-specific reporting and appraisal extensions (eg, PRISMA-LLM [Preferred Reporting Items for Systematic Reviews and Meta-Analyses–large language model] and AMSTAR-LLM [A Measurement Tool to Assess Systematic Reviews-large language model]).

Subjects

MeSH terms (MEDLINE)

Systematic Reviews as TopicArtificial IntelligenceResearch DesignHumansReproducibility of ResultsMeta-Analysis as TopicLarge Language Models

Author keywords

AMSTAR-2PRISMA 2020artificial intelligenceevidence synthesislarge language modelsmeta-researchreproducibilitysystematic reviewtransparency

Authors and institutions

  • Department of Medicine, Division of Endocrinology, University of Toronto, 2075 Bayview Avenue, Toronto, ON, M4N 3M5, Canada, 1 416-480-6705, 1 416-480-5761.
  • Institute of Health Policy, Management and Evaluation (IHPME), Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada.
  • Faculty of Medicine, University of Toronto, Toronto, ON, Canada.
  • Department of Medicine, Division of Geriatric Medicine, University of Toronto, Toronto, ON, Canada.
  • Knowledge Translation Program, Li Ka Shing Knowledge Institute, University of Toronto, Toronto, ON, Canada.
  • Library Services, Sunnybrook Health Science Centre, Toronto, ON, Canada.

ORCID profiles: Mohammad Jay, Mary Morgan, Sharon Elizabeth Straus, Emma Wilson, Rinku Sutradhar, Catherine Yu, Lesley Gotlib Conn, Christoffer Dharma, Lorraine Lipscombe, Antoine Eskander

Cite

APA

Jay, M., Morgan, M., Straus, S. E., Wilson, E., Sutradhar, R., Yu, C., Gotlib Conn, L., Dharma, C., Lipscombe, L., & Eskander, A. (2026). Evaluating the Methodological Quality of Artificial Intelligence–Assisted Systematic Reviews: Protocol for a Mixed Methods Meta-Research Study. JMIR Research Protocols, 15, e90588-e90588. https://doi.org/10.2196/90588

Vancouver

Jay M, Morgan M, Straus SE, Wilson E, Sutradhar R, Yu C, et al. Evaluating the Methodological Quality of Artificial Intelligence–Assisted Systematic Reviews: Protocol for a Mixed Methods Meta-Research Study. JMIR Research Protocols. 2026;15:e90588-e90588. doi:10.2196/90588

BibTeX
@article{Jay2026,
  author = {Jay, Mohammad and Morgan, Mary and Straus, Sharon Elizabeth and Wilson, Emma and Sutradhar, Rinku and Yu, Catherine and Gotlib Conn, Lesley and Dharma, Christoffer and Lipscombe, Lorraine and Eskander, Antoine},
  title = {Evaluating the Methodological Quality of Artificial Intelligence–Assisted Systematic Reviews: Protocol for a Mixed Methods Meta-Research Study},
  journal = {JMIR Research Protocols},
  year = {2026},
  volume = {15},
  pages = {e90588--e90588},
  doi = {10.2196/90588}
}
Start for free →