Performance and Consistency of Large Language Models in Key Labor‐Intensive Tasks of Systematic Reviews
Journal of Evaluation in Clinical Practice · 2026
DOI: 10.1111/jep.70596 ↗At a glance
Journal article in Journal of Evaluation in Clinical Practice, 2026. Written by Yi-Ran Liu and 6 co-authors. Institution: The First Clinical College, Chongqing Medical University, Chongqing, China. Major MEDLINE topics: Large Language Models, Systematic Reviews as Topic. Cited 1 times so far (Crossref); 27 references. The full text is freely available in PubMed Central. Funded by: Chongqing Municipal Health Commission.
Abstract (original language)
Objective
To evaluate the performance and consistency of Large Language Models (LLMs) in core systematic review (SR) tasks and to introduce open‐source tools for automated batch processing that provide decision rationales.
Methods
We assessed GPT‐4o, Kimi‐K2, DeepSeek‐V3, and DeepSeek‐R1 on five SR tasks: title/abstract screening (3550 records), full‐text screening (233 texts), data extraction (112 RCTs), Risk of Bias (ROB) assessment (112 RCTs), and AMSTAR‐2 assessment (20 SRs). Each model was evaluated twice to measure consistency. All outputs required supporting rationales and verbatim evidence.
Results
LLMs demonstrated proficiency across tasks, with generally high intra‐model but lower inter‐model consistency. In screening, models showed lower precision (0.27–0.40) but high recall (0.83–0.91) and specificity (0.83–0.91). DeepSeek‐R1 and DeepSeek‐V3 excelled in title/abstract and full‐text screening, respectively. Data extraction accuracy was similar across models (0.78–0.82). Kimi‐K2 achieved the highest ROB F1 score (0.71). AMSTAR‐2 assessments were generally acceptable.
Discussion
While effective, LLMs showed variable performance across SR tasks. The mandatory output of rationales and evidence enhances transparency and allows for human verification of AI decisions.
Conclusion
We provide a suite of automated tools for key SR tasks. By leveraging these tools to validate model outputs rather than starting manually, reviewers can significantly improve workflow efficiency while maintaining methodological rigour.
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- The First Clinical College, Chongqing Medical University, Chongqing, China.
- Mental Health Center, University-Town Hospital of Chongqing Medical University, Chongqing, China.
ORCID profiles: Dong-Dong Zhou
Funding
- Chongqing Municipal Health Commission
Cite
APA
Liu, Y. R., Wang, X. L., Zhou, Z. X., He, Y. J., Li, Y. Z., Liu, C., & Zhou, D. D. (2026). Performance and Consistency of Large Language Models in Key Labor‐Intensive Tasks of Systematic Reviews. Journal of Evaluation in Clinical Practice. https://doi.org/10.1111/jep.70596
Vancouver
Liu YR, Wang XL, Zhou ZX, He YJ, Li YZ, Liu C, et al. Performance and Consistency of Large Language Models in Key Labor‐Intensive Tasks of Systematic Reviews. Journal of Evaluation in Clinical Practice. 2026. doi:10.1111/jep.70596
BibTeX
@article{Liu2026,
author = {Liu, Yi-Ran and Wang, Xi-Ling and Zhou, Zi-Xuan and He, Yuan-Ji and Li, Yu-Zhang and Liu, Chuan and Zhou, Dong-Dong},
title = {Performance and Consistency of Large Language Models in Key Labor‐Intensive Tasks of Systematic Reviews},
journal = {Journal of Evaluation in Clinical Practice},
year = {2026},
doi = {10.1111/jep.70596}
}
EUVIMED