EUVIMEDOpdag EUVIMED
Open access

Assessing the methodologic quality of systematic reviews using generative large language models

Bowen Yao, Onuralp Ergun, Maylynn Ding, Carly D Miller, Vikram M Narayan, Philipp Dahm

Canadian Urological Association Journal · 2025-08-28 · 19 (12)

DOI: 10.5489/cuaj.9243 ↗

At a glance

Journal article in Canadian Urological Association Journal, published online on 2025-08-28. Written by Bowen Yao and 5 co-authors. Institution: Minneapolis VA Healthcare System, Minneapolis, MN, United States. Cited 2 times so far (Crossref). The full text is freely available in PubMed Central.

AbstractBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Introduction

We aimed to evaluate whether generative large language models (LLMs) can accurately assess the methodologic quality of systematic reviews (SRs).

Methods

A total of 114 SRs from five leading urology journals were included in the study. Human reviewers graded each of the SRs in duplicates, with differences adjudicated by a third expert. We created a customized GPT “Urology AMSTAR 2 Quality Assessor” and graded the 114 SRs in three iterations using a zero-shot method. We performed an enhanced trial focusing on critical criteria by giving GPT detailed, step-by-step instructions for each of the SRs using chain-of-thought method. Accuracy, sensitivity, specificity, and F1 score for each GPT trial was calculated against human results. Internal validity among three trials were computed.

Results

GPT had an overall congruence of 75%, with 77% in critical criteria and 73% in non-critical criteria when compared to human results. The average F1 score was 0.66. There was a high internal validity at 85% among three iterations. GPT accurately assigned 89% of studies into the correct overall category. When given specific, step-by-step instructions, congruence of critical criteria improved to 91%, and overall quality assessment accuracy to 93%.

Conclusions

GPT showed promising ability to efficiently and accurately assess the quality of SRs in urology.

Authors and institutions

  • Minneapolis VA Healthcare System, Minneapolis, MN, United States.
  • Department of Urology, University of Minnesota, Minneapolis, MN, United States.
  • Department of Urology, Emory University, Atlanta, GA, United States.

Cite

APA

Yao, B., Ergun, O., Ding, M., Miller, C. D., Narayan, V. M., & Dahm, P. (2025). Assessing the methodologic quality of systematic reviews using generative large language models. Canadian Urological Association Journal, 19(12). https://doi.org/10.5489/cuaj.9243

Vancouver

Yao B, Ergun O, Ding M, Miller CD, Narayan VM, Dahm P. Assessing the methodologic quality of systematic reviews using generative large language models. Canadian Urological Association Journal. 2025;19(12). doi:10.5489/cuaj.9243

BibTeX
@article{Yao2025,
  author = {Yao, Bowen and Ergun, Onuralp and Ding, Maylynn and Miller, Carly D and Narayan, Vikram M and Dahm, Philipp},
  title = {Assessing the methodologic quality of systematic reviews using generative large language models},
  journal = {Canadian Urological Association Journal},
  year = {2025},
  volume = {19},
  number = {12},
  doi = {10.5489/cuaj.9243}
}
Start for free →