EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access

Automated analyses of risk of bias and critical appraisal of systematic reviews (ROBIS and AMSTAR 2): a comparison of the performance of 4 large language models

Diego A Forero iD, Sandra E Abreu, Blanca E Tovar, Marilyn H Oermann

Journal of the American Medical Informatics Association · 01.09.2025 · 32 (9) · 1471-1476

DOI: 10.1093/jamia/ocaf117 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 18.07.2025. Verfasst von Diego A Forero und 3 weiteren Autor:innen. Beteiligte Einrichtung: School of Health and Sport Sciences, Fundación Universitaria del Área Andina, Bogotá, 110231, Colombia. Hauptthemen laut MEDLINE: Systematic Reviews as Topic, Meta-Analysis as Topic, Natural Language Processing. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 46 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: Minciencias and Areandina.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

To explore the performance of 4 large language model (LLM) chatbots for the analysis of 2 of the most commonly used tools for the advanced analysis of systematic reviews (SRs) and meta-analyses. Materials and Methods We explored the performance of 4 LLM chatbots (ChatGPT, Gemini, DeepSeek, and QWEN) for the analysis of ROBIS and AMSTAR 2 tools (sample sizes: 20 SRs), in comparison with assessments by human experts.

Results

Gemini showed the best agreement with human experts for both ROBIS and AMSTAR 2 (accuracy: 58% and 70%). The second best LLM chatbots were ChatGPT and QWEN, for ROBIS and AMSTAR 2, respectively.

Discussion

Some LLM chatbots underestimated the risk of bias or overestimated the confidence of the results in published SRs, which is compatible with recent articles for other tools.

Conclusion

This is one of the first studies comparing the performance of several LLM chatbots for the automated analyses of ROBIS and AMSTAR 2.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Systematic Reviews as TopicMeta-Analysis as TopicNatural Language ProcessingHumansBiasLanguageLarge Language Models

Stichwörter der Autor:innen

evidence-based medicinegenerative artificial intelligencemeta-analysisrisk of biassystematic reviews

Autor:innen und Institutionen

  • School of Health and Sport Sciences, Fundación Universitaria del Área Andina, Bogotá, 110231, Colombia.
  • Psychology Program, Fundación Universitaria del Área Andina, Medellín, 050005, Colombia.
  • Nursing Program, School of Health and Sport Sciences, Fundación Universitaria del Área Andina, Bogotá, 110231, Colombia.
  • School of Nursing, Duke University School of Nursing, 27710, United States.

ORCID-Profile: Diego A Forero

Förderung

  • Minciencias and Areandina

Zitieren

APA

Forero, D. A., Abreu, S. E., Tovar, B. E., & Oermann, M. H. (2025). Automated analyses of risk of bias and critical appraisal of systematic reviews (ROBIS and AMSTAR 2): a comparison of the performance of 4 large language models. Journal of the American Medical Informatics Association, 32(9), 1471-1476. https://doi.org/10.1093/jamia/ocaf117

Vancouver

Forero DA, Abreu SE, Tovar BE, Oermann MH. Automated analyses of risk of bias and critical appraisal of systematic reviews (ROBIS and AMSTAR 2): a comparison of the performance of 4 large language models. Journal of the American Medical Informatics Association. 2025;32(9):1471-1476. doi:10.1093/jamia/ocaf117

BibTeX
@article{Forero2025,
  author = {Forero, Diego A and Abreu, Sandra E and Tovar, Blanca E and Oermann, Marilyn H},
  title = {Automated analyses of risk of bias and critical appraisal of systematic reviews (ROBIS and AMSTAR 2): a comparison of the performance of 4 large language models},
  journal = {Journal of the American Medical Informatics Association},
  year = {2025},
  volume = {32},
  number = {9},
  pages = {1471--1476},
  doi = {10.1093/jamia/ocaf117}
}
Kostenlos starten →