Assessing the methodologic quality of systematic reviews using generative large language models
Canadian Urological Association Journal · 2025-08-28 · 19 (12)
DOI: 10.5489/cuaj.9243 ↗At a glance
Journal article in Canadian Urological Association Journal, published online on 2025-08-28. Written by Bowen Yao and 5 co-authors. Institution: Minneapolis VA Healthcare System, Minneapolis, MN, United States. Cited 2 times so far (Crossref). The full text is freely available in PubMed Central.
Abstract (original language)
Introduction
We aimed to evaluate whether generative large language models (LLMs) can accurately assess the methodologic quality of systematic reviews (SRs).
Methods
A total of 114 SRs from five leading urology journals were included in the study. Human reviewers graded each of the SRs in duplicates, with differences adjudicated by a third expert. We created a customized GPT “Urology AMSTAR 2 Quality Assessor” and graded the 114 SRs in three iterations using a zero-shot method. We performed an enhanced trial focusing on critical criteria by giving GPT detailed, step-by-step instructions for each of the SRs using chain-of-thought method. Accuracy, sensitivity, specificity, and F1 score for each GPT trial was calculated against human results. Internal validity among three trials were computed.
Results
GPT had an overall congruence of 75%, with 77% in critical criteria and 73% in non-critical criteria when compared to human results. The average F1 score was 0.66. There was a high internal validity at 85% among three iterations. GPT accurately assigned 89% of studies into the correct overall category. When given specific, step-by-step instructions, congruence of critical criteria improved to 91%, and overall quality assessment accuracy to 93%.
Conclusions
GPT showed promising ability to efficiently and accurately assess the quality of SRs in urology.
Authors and institutions
- Minneapolis VA Healthcare System, Minneapolis, MN, United States.
- Department of Urology, University of Minnesota, Minneapolis, MN, United States.
- Department of Urology, Emory University, Atlanta, GA, United States.
Cite
APA
Yao, B., Ergun, O., Ding, M., Miller, C. D., Narayan, V. M., & Dahm, P. (2025). Assessing the methodologic quality of systematic reviews using generative large language models. Canadian Urological Association Journal, 19(12). https://doi.org/10.5489/cuaj.9243
Vancouver
Yao B, Ergun O, Ding M, Miller CD, Narayan VM, Dahm P. Assessing the methodologic quality of systematic reviews using generative large language models. Canadian Urological Association Journal. 2025;19(12). doi:10.5489/cuaj.9243
BibTeX
@article{Yao2025,
author = {Yao, Bowen and Ergun, Onuralp and Ding, Maylynn and Miller, Carly D and Narayan, Vikram M and Dahm, Philipp},
title = {Assessing the methodologic quality of systematic reviews using generative large language models},
journal = {Canadian Urological Association Journal},
year = {2025},
volume = {19},
number = {12},
doi = {10.5489/cuaj.9243}
}
EUVIMED