EUVIMEDEUVIMED entdecken
Open Access

Performance and Consistency of Large Language Models in Key Labor‐Intensive Tasks of Systematic Reviews

Yi-Ran Liu, Xi-Ling Wang, Zi-Xuan Zhou, Yuan-Ji He, Yu-Zhang Li, Chuan Liu, Dong-Dong Zhou iD

Journal of Evaluation in Clinical Practice · 2026

DOI: 10.1111/jep.70596 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Evaluation in Clinical Practice, 2026. Verfasst von Yi-Ran Liu und 6 weiteren Autor:innen. Beteiligte Einrichtung: The First Clinical College, Chongqing Medical University, Chongqing, China. Hauptthemen laut MEDLINE: Large Language Models, Systematic Reviews as Topic. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 27 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: Chongqing Municipal Health Commission.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

To evaluate the performance and consistency of Large Language Models (LLMs) in core systematic review (SR) tasks and to introduce open‐source tools for automated batch processing that provide decision rationales.

Methods

We assessed GPT‐4o, Kimi‐K2, DeepSeek‐V3, and DeepSeek‐R1 on five SR tasks: title/abstract screening (3550 records), full‐text screening (233 texts), data extraction (112 RCTs), Risk of Bias (ROB) assessment (112 RCTs), and AMSTAR‐2 assessment (20 SRs). Each model was evaluated twice to measure consistency. All outputs required supporting rationales and verbatim evidence.

Results

LLMs demonstrated proficiency across tasks, with generally high intra‐model but lower inter‐model consistency. In screening, models showed lower precision (0.27–0.40) but high recall (0.83–0.91) and specificity (0.83–0.91). DeepSeek‐R1 and DeepSeek‐V3 excelled in title/abstract and full‐text screening, respectively. Data extraction accuracy was similar across models (0.78–0.82). Kimi‐K2 achieved the highest ROB F1 score (0.71). AMSTAR‐2 assessments were generally acceptable.

Discussion

While effective, LLMs showed variable performance across SR tasks. The mandatory output of rationales and evidence enhances transparency and allows for human verification of AI decisions.

Conclusion

We provide a suite of automated tools for key SR tasks. By leveraging these tools to validate model outputs rather than starting manually, reviewers can significantly improve workflow efficiency while maintaining methodological rigour.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsSystematic Reviews as TopicHumansReproducibility of Results

Stichwörter der Autor:innen

automationevidence‐based medicinelarge language modelsrisk of biassystematic reviews

Autor:innen und Institutionen

  • The First Clinical College, Chongqing Medical University, Chongqing, China.
  • Mental Health Center, University-Town Hospital of Chongqing Medical University, Chongqing, China.

ORCID-Profile: Dong-Dong Zhou

Förderung

  • Chongqing Municipal Health Commission

Zitieren

APA

Liu, Y. R., Wang, X. L., Zhou, Z. X., He, Y. J., Li, Y. Z., Liu, C., & Zhou, D. D. (2026). Performance and Consistency of Large Language Models in Key Labor‐Intensive Tasks of Systematic Reviews. Journal of Evaluation in Clinical Practice. https://doi.org/10.1111/jep.70596

Vancouver

Liu YR, Wang XL, Zhou ZX, He YJ, Li YZ, Liu C, et al. Performance and Consistency of Large Language Models in Key Labor‐Intensive Tasks of Systematic Reviews. Journal of Evaluation in Clinical Practice. 2026. doi:10.1111/jep.70596

BibTeX
@article{Liu2026,
  author = {Liu, Yi-Ran and Wang, Xi-Ling and Zhou, Zi-Xuan and He, Yuan-Ji and Li, Yu-Zhang and Liu, Chuan and Zhou, Dong-Dong},
  title = {Performance and Consistency of Large Language Models in Key Labor‐Intensive Tasks of Systematic Reviews},
  journal = {Journal of Evaluation in Clinical Practice},
  year = {2026},
  doi = {10.1111/jep.70596}
}
Kostenlos starten →