Performance and Consistency of Large Language Models in Key Labor‐Intensive Tasks of Systematic Reviews
Journal of Evaluation in Clinical Practice · 2026
DOI: 10.1111/jep.70596 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Evaluation in Clinical Practice, 2026. Verfasst von Yi-Ran Liu und 6 weiteren Autor:innen. Beteiligte Einrichtung: The First Clinical College, Chongqing Medical University, Chongqing, China. Hauptthemen laut MEDLINE: Large Language Models, Systematic Reviews as Topic. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 27 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: Chongqing Municipal Health Commission.
Abstract (Originalsprache)
Objective
To evaluate the performance and consistency of Large Language Models (LLMs) in core systematic review (SR) tasks and to introduce open‐source tools for automated batch processing that provide decision rationales.
Methods
We assessed GPT‐4o, Kimi‐K2, DeepSeek‐V3, and DeepSeek‐R1 on five SR tasks: title/abstract screening (3550 records), full‐text screening (233 texts), data extraction (112 RCTs), Risk of Bias (ROB) assessment (112 RCTs), and AMSTAR‐2 assessment (20 SRs). Each model was evaluated twice to measure consistency. All outputs required supporting rationales and verbatim evidence.
Results
LLMs demonstrated proficiency across tasks, with generally high intra‐model but lower inter‐model consistency. In screening, models showed lower precision (0.27–0.40) but high recall (0.83–0.91) and specificity (0.83–0.91). DeepSeek‐R1 and DeepSeek‐V3 excelled in title/abstract and full‐text screening, respectively. Data extraction accuracy was similar across models (0.78–0.82). Kimi‐K2 achieved the highest ROB F1 score (0.71). AMSTAR‐2 assessments were generally acceptable.
Discussion
While effective, LLMs showed variable performance across SR tasks. The mandatory output of rationales and evidence enhances transparency and allows for human verification of AI decisions.
Conclusion
We provide a suite of automated tools for key SR tasks. By leveraging these tools to validate model outputs rather than starting manually, reviewers can significantly improve workflow efficiency while maintaining methodological rigour.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- The First Clinical College, Chongqing Medical University, Chongqing, China.
- Mental Health Center, University-Town Hospital of Chongqing Medical University, Chongqing, China.
ORCID-Profile: Dong-Dong Zhou
Förderung
- Chongqing Municipal Health Commission
Zitieren
APA
Liu, Y. R., Wang, X. L., Zhou, Z. X., He, Y. J., Li, Y. Z., Liu, C., & Zhou, D. D. (2026). Performance and Consistency of Large Language Models in Key Labor‐Intensive Tasks of Systematic Reviews. Journal of Evaluation in Clinical Practice. https://doi.org/10.1111/jep.70596
Vancouver
Liu YR, Wang XL, Zhou ZX, He YJ, Li YZ, Liu C, et al. Performance and Consistency of Large Language Models in Key Labor‐Intensive Tasks of Systematic Reviews. Journal of Evaluation in Clinical Practice. 2026. doi:10.1111/jep.70596
BibTeX
@article{Liu2026,
author = {Liu, Yi-Ran and Wang, Xi-Ling and Zhou, Zi-Xuan and He, Yuan-Ji and Li, Yu-Zhang and Liu, Chuan and Zhou, Dong-Dong},
title = {Performance and Consistency of Large Language Models in Key Labor‐Intensive Tasks of Systematic Reviews},
journal = {Journal of Evaluation in Clinical Practice},
year = {2026},
doi = {10.1111/jep.70596}
}
EUVIMED