Large language model-based paper classification framework with key-insight extraction and confidence-weighted voting
Research Synthesis Methods · 2026
DOI: 10.1017/rsm.2026.10094 ↗Kurz gefasst
Fachartikel in der Zeitschrift Research Synthesis Methods, 2026. Verfasst von Zihan Song und 9 weiteren Autor:innen. Beteiligte Einrichtung: College of Industrial Internet, Hubei Engineering Institute, China. Hauptthemen laut MEDLINE: Large Language Models, Systematic Reviews as Topic. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 46 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Systematic reviews (SRs) are critical for evidence-based research but are time-consuming and labor-intensive. The rapid expansion of academic publications further challenges the performance and applicability of existing screening and classification methods. While large language models (LLMs) present new opportunities for automation, limited research has examined whether they can achieve classification performance comparable to human reviewers in large-scale, multi-class settings. With the goal of improving classification performance, we proposed an LLM-based framework that leverages full-text key-insight extraction to enhance literature classification. We constructed a manually curated dataset of 900 articles from 17 published SRs to quantitatively evaluate the classification capabilities of LLMs. The results provided empirical evidence of LLMs’ potential in supporting large-scale SRs and introduced a practical pathway for improving efficiency and reliability in evidence synthesis. Empirical results showed that key-insight-based classification (KBC) significantly outperforms abstract-based classification (ABC). We implemented a confidence-weighted voting (CWV) mechanism using multiple LLMs to improve robustness. The CWV method achieved the highest macro F 1-score of 0.796, substantially exceeding KBC (0.732), ABC (0.676), and unsupervised K-means clustering (0.446). By employing zero-shot LLMs, our approach demonstrated the potential for enhanced adaptability across diverse domains and classification tasks without requiring fine-tuning, demonstrating that a carefully designed pipeline can enable LLMs to achieve classification performance comparable to human reviewers.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- College of Industrial Internet, Hubei Engineering Institute, China.
- Research and Development, Center for Emergency Medicine Advancement, Nepal.
- Department of Management Information Systems, Dong-A University, Republic of Korea.
- Department of External Cooperation, Zhejiang Academy of Industrial Internet Development, China.
Zitieren
APA
Song, Z., Huang, S., Thapa, N., Zhang, X., Park, B. K., Lu, J., Li, W., Liu, W., Zhan, B., & Li, J. (2026). Large language model-based paper classification framework with key-insight extraction and confidence-weighted voting. Research Synthesis Methods. https://doi.org/10.1017/rsm.2026.10094
Vancouver
Song Z, Huang S, Thapa N, Zhang X, Park BK, Lu J, et al. Large language model-based paper classification framework with key-insight extraction and confidence-weighted voting. Research Synthesis Methods. 2026. doi:10.1017/rsm.2026.10094
BibTeX
@article{Song2026,
author = {Song, Zihan and Huang, Shan and Thapa, Ngeemasara and Zhang, Xin and Park, Byung-Kwon and Lu, Jie and Li, Wenyang and Liu, Wenbin and Zhan, Bei and Li, Jianfei},
title = {Large language model-based paper classification framework with key-insight extraction and confidence-weighted voting},
journal = {Research Synthesis Methods},
year = {2026},
doi = {10.1017/rsm.2026.10094}
}
EUVIMED