EUVIMEDOpdag EUVIMED
Open access

Performance and Consistency of Large Language Models in Key Labor‐Intensive Tasks of Systematic Reviews

Yi-Ran Liu, Xi-Ling Wang, Zi-Xuan Zhou, Yuan-Ji He, Yu-Zhang Li, Chuan Liu, Dong-Dong Zhou iD

Journal of Evaluation in Clinical Practice · 2026

DOI: 10.1111/jep.70596 ↗

At a glance

Journal article in Journal of Evaluation in Clinical Practice, 2026. Written by Yi-Ran Liu and 6 co-authors. Institution: The First Clinical College, Chongqing Medical University, Chongqing, China. Major MEDLINE topics: Large Language Models, Systematic Reviews as Topic. Cited 1 times so far (Crossref); 27 references. The full text is freely available in PubMed Central. Funded by: Chongqing Municipal Health Commission.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Objective

To evaluate the performance and consistency of Large Language Models (LLMs) in core systematic review (SR) tasks and to introduce open‐source tools for automated batch processing that provide decision rationales.

Methods

We assessed GPT‐4o, Kimi‐K2, DeepSeek‐V3, and DeepSeek‐R1 on five SR tasks: title/abstract screening (3550 records), full‐text screening (233 texts), data extraction (112 RCTs), Risk of Bias (ROB) assessment (112 RCTs), and AMSTAR‐2 assessment (20 SRs). Each model was evaluated twice to measure consistency. All outputs required supporting rationales and verbatim evidence.

Results

LLMs demonstrated proficiency across tasks, with generally high intra‐model but lower inter‐model consistency. In screening, models showed lower precision (0.27–0.40) but high recall (0.83–0.91) and specificity (0.83–0.91). DeepSeek‐R1 and DeepSeek‐V3 excelled in title/abstract and full‐text screening, respectively. Data extraction accuracy was similar across models (0.78–0.82). Kimi‐K2 achieved the highest ROB F1 score (0.71). AMSTAR‐2 assessments were generally acceptable.

Discussion

While effective, LLMs showed variable performance across SR tasks. The mandatory output of rationales and evidence enhances transparency and allows for human verification of AI decisions.

Conclusion

We provide a suite of automated tools for key SR tasks. By leveraging these tools to validate model outputs rather than starting manually, reviewers can significantly improve workflow efficiency while maintaining methodological rigour.

Subjects

MeSH terms (MEDLINE)

Large Language ModelsSystematic Reviews as TopicHumansReproducibility of Results

Author keywords

automationevidence‐based medicinelarge language modelsrisk of biassystematic reviews

Authors and institutions

  • The First Clinical College, Chongqing Medical University, Chongqing, China.
  • Mental Health Center, University-Town Hospital of Chongqing Medical University, Chongqing, China.

ORCID profiles: Dong-Dong Zhou

Funding

  • Chongqing Municipal Health Commission

Cite

APA

Liu, Y. R., Wang, X. L., Zhou, Z. X., He, Y. J., Li, Y. Z., Liu, C., & Zhou, D. D. (2026). Performance and Consistency of Large Language Models in Key Labor‐Intensive Tasks of Systematic Reviews. Journal of Evaluation in Clinical Practice. https://doi.org/10.1111/jep.70596

Vancouver

Liu YR, Wang XL, Zhou ZX, He YJ, Li YZ, Liu C, et al. Performance and Consistency of Large Language Models in Key Labor‐Intensive Tasks of Systematic Reviews. Journal of Evaluation in Clinical Practice. 2026. doi:10.1111/jep.70596

BibTeX
@article{Liu2026,
  author = {Liu, Yi-Ran and Wang, Xi-Ling and Zhou, Zi-Xuan and He, Yuan-Ji and Li, Yu-Zhang and Liu, Chuan and Zhou, Dong-Dong},
  title = {Performance and Consistency of Large Language Models in Key Labor‐Intensive Tasks of Systematic Reviews},
  journal = {Journal of Evaluation in Clinical Practice},
  year = {2026},
  doi = {10.1111/jep.70596}
}
Start for free →