Automated analyses of risk of bias and critical appraisal of systematic reviews (ROBIS and AMSTAR 2): a comparison of the performance of 4 large language models
Journal of the American Medical Informatics Association · 01.09.2025 · 32 (9) · 1471-1476
DOI: 10.1093/jamia/ocaf117 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 18.07.2025. Verfasst von Diego A Forero und 3 weiteren Autor:innen. Beteiligte Einrichtung: School of Health and Sport Sciences, Fundación Universitaria del Área Andina, Bogotá, 110231, Colombia. Hauptthemen laut MEDLINE: Systematic Reviews as Topic, Meta-Analysis as Topic, Natural Language Processing. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 46 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: Minciencias and Areandina.
Abstract (Originalsprache)
Objectives
To explore the performance of 4 large language model (LLM) chatbots for the analysis of 2 of the most commonly used tools for the advanced analysis of systematic reviews (SRs) and meta-analyses. Materials and Methods We explored the performance of 4 LLM chatbots (ChatGPT, Gemini, DeepSeek, and QWEN) for the analysis of ROBIS and AMSTAR 2 tools (sample sizes: 20 SRs), in comparison with assessments by human experts.
Results
Gemini showed the best agreement with human experts for both ROBIS and AMSTAR 2 (accuracy: 58% and 70%). The second best LLM chatbots were ChatGPT and QWEN, for ROBIS and AMSTAR 2, respectively.
Discussion
Some LLM chatbots underestimated the risk of bias or overestimated the confidence of the results in published SRs, which is compatible with recent articles for other tools.
Conclusion
This is one of the first studies comparing the performance of several LLM chatbots for the automated analyses of ROBIS and AMSTAR 2.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- School of Health and Sport Sciences, Fundación Universitaria del Área Andina, Bogotá, 110231, Colombia.
- Psychology Program, Fundación Universitaria del Área Andina, Medellín, 050005, Colombia.
- Nursing Program, School of Health and Sport Sciences, Fundación Universitaria del Área Andina, Bogotá, 110231, Colombia.
- School of Nursing, Duke University School of Nursing, 27710, United States.
ORCID-Profile: Diego A Forero
Förderung
- Minciencias and Areandina
Zitieren
APA
Forero, D. A., Abreu, S. E., Tovar, B. E., & Oermann, M. H. (2025). Automated analyses of risk of bias and critical appraisal of systematic reviews (ROBIS and AMSTAR 2): a comparison of the performance of 4 large language models. Journal of the American Medical Informatics Association, 32(9), 1471-1476. https://doi.org/10.1093/jamia/ocaf117
Vancouver
Forero DA, Abreu SE, Tovar BE, Oermann MH. Automated analyses of risk of bias and critical appraisal of systematic reviews (ROBIS and AMSTAR 2): a comparison of the performance of 4 large language models. Journal of the American Medical Informatics Association. 2025;32(9):1471-1476. doi:10.1093/jamia/ocaf117
BibTeX
@article{Forero2025,
author = {Forero, Diego A and Abreu, Sandra E and Tovar, Blanca E and Oermann, Marilyn H},
title = {Automated analyses of risk of bias and critical appraisal of systematic reviews (ROBIS and AMSTAR 2): a comparison of the performance of 4 large language models},
journal = {Journal of the American Medical Informatics Association},
year = {2025},
volume = {32},
number = {9},
pages = {1471--1476},
doi = {10.1093/jamia/ocaf117}
}
EUVIMED