Assessing the methodologic quality of systematic reviews using generative large language models
Canadian Urological Association Journal · 28.08.2025 · 19 (12)
DOI: 10.5489/cuaj.9243 ↗Kurz gefasst
Fachartikel in der Zeitschrift Canadian Urological Association Journal, online erschienen am 28.08.2025. Verfasst von Bowen Yao und 5 weiteren Autor:innen. Beteiligte Einrichtung: Minneapolis VA Healthcare System, Minneapolis, MN, United States. Die Arbeit wurde bisher 2-mal zitiert (Crossref). Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Introduction
We aimed to evaluate whether generative large language models (LLMs) can accurately assess the methodologic quality of systematic reviews (SRs).
Methods
A total of 114 SRs from five leading urology journals were included in the study. Human reviewers graded each of the SRs in duplicates, with differences adjudicated by a third expert. We created a customized GPT “Urology AMSTAR 2 Quality Assessor” and graded the 114 SRs in three iterations using a zero-shot method. We performed an enhanced trial focusing on critical criteria by giving GPT detailed, step-by-step instructions for each of the SRs using chain-of-thought method. Accuracy, sensitivity, specificity, and F1 score for each GPT trial was calculated against human results. Internal validity among three trials were computed.
Results
GPT had an overall congruence of 75%, with 77% in critical criteria and 73% in non-critical criteria when compared to human results. The average F1 score was 0.66. There was a high internal validity at 85% among three iterations. GPT accurately assigned 89% of studies into the correct overall category. When given specific, step-by-step instructions, congruence of critical criteria improved to 91%, and overall quality assessment accuracy to 93%.
Conclusions
GPT showed promising ability to efficiently and accurately assess the quality of SRs in urology.
Autor:innen und Institutionen
- Minneapolis VA Healthcare System, Minneapolis, MN, United States.
- Department of Urology, University of Minnesota, Minneapolis, MN, United States.
- Department of Urology, Emory University, Atlanta, GA, United States.
Zitieren
APA
Yao, B., Ergun, O., Ding, M., Miller, C. D., Narayan, V. M., & Dahm, P. (2025). Assessing the methodologic quality of systematic reviews using generative large language models. Canadian Urological Association Journal, 19(12). https://doi.org/10.5489/cuaj.9243
Vancouver
Yao B, Ergun O, Ding M, Miller CD, Narayan VM, Dahm P. Assessing the methodologic quality of systematic reviews using generative large language models. Canadian Urological Association Journal. 2025;19(12). doi:10.5489/cuaj.9243
BibTeX
@article{Yao2025,
author = {Yao, Bowen and Ergun, Onuralp and Ding, Maylynn and Miller, Carly D and Narayan, Vikram M and Dahm, Philipp},
title = {Assessing the methodologic quality of systematic reviews using generative large language models},
journal = {Canadian Urological Association Journal},
year = {2025},
volume = {19},
number = {12},
doi = {10.5489/cuaj.9243}
}
EUVIMED