EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Large language model-based paper classification framework with key-insight extraction and confidence-weighted voting

Zihan Song, Shan Huang, Ngeemasara Thapa, Xin Zhang, Byung-Kwon Park, Jie Lu, Wenyang Li, Wenbin Liu, Bei Zhan, Jianfei Li

Research Synthesis Methods · 2026

DOI: 10.1017/rsm.2026.10094 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Research Synthesis Methods, 2026. Verfasst von Zihan Song und 9 weiteren Autor:innen. Beteiligte Einrichtung: College of Industrial Internet, Hubei Engineering Institute, China. Hauptthemen laut MEDLINE: Large Language Models, Systematic Reviews as Topic. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 46 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Systematic reviews (SRs) are critical for evidence-based research but are time-consuming and labor-intensive. The rapid expansion of academic publications further challenges the performance and applicability of existing screening and classification methods. While large language models (LLMs) present new opportunities for automation, limited research has examined whether they can achieve classification performance comparable to human reviewers in large-scale, multi-class settings. With the goal of improving classification performance, we proposed an LLM-based framework that leverages full-text key-insight extraction to enhance literature classification. We constructed a manually curated dataset of 900 articles from 17 published SRs to quantitatively evaluate the classification capabilities of LLMs. The results provided empirical evidence of LLMs’ potential in supporting large-scale SRs and introduced a practical pathway for improving efficiency and reliability in evidence synthesis. Empirical results showed that key-insight-based classification (KBC) significantly outperforms abstract-based classification (ABC). We implemented a confidence-weighted voting (CWV) mechanism using multiple LLMs to improve robustness. The CWV method achieved the highest macro F 1-score of 0.796, substantially exceeding KBC (0.732), ABC (0.676), and unsupervised K-means clustering (0.446). By employing zero-shot LLMs, our approach demonstrated the potential for enhanced adaptability across diverse domains and classification tasks without requiring fine-tuning, demonstrating that a carefully designed pipeline can enable LLMs to achieve classification performance comparable to human reviewers.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsSystematic Reviews as TopicHumansReproducibility of ResultsAlgorithmsClassification AlgorithmsVotingPaperClustering Algorithms

Stichwörter der Autor:innen

artificial intelligenceevidence synthesislarge language modelliterature screeningpaper classificationsystematic review

Autor:innen und Institutionen

  • College of Industrial Internet, Hubei Engineering Institute, China.
  • Research and Development, Center for Emergency Medicine Advancement, Nepal.
  • Department of Management Information Systems, Dong-A University, Republic of Korea.
  • Department of External Cooperation, Zhejiang Academy of Industrial Internet Development, China.

Zitieren

APA

Song, Z., Huang, S., Thapa, N., Zhang, X., Park, B. K., Lu, J., Li, W., Liu, W., Zhan, B., & Li, J. (2026). Large language model-based paper classification framework with key-insight extraction and confidence-weighted voting. Research Synthesis Methods. https://doi.org/10.1017/rsm.2026.10094

Vancouver

Song Z, Huang S, Thapa N, Zhang X, Park BK, Lu J, et al. Large language model-based paper classification framework with key-insight extraction and confidence-weighted voting. Research Synthesis Methods. 2026. doi:10.1017/rsm.2026.10094

BibTeX
@article{Song2026,
  author = {Song, Zihan and Huang, Shan and Thapa, Ngeemasara and Zhang, Xin and Park, Byung-Kwon and Lu, Jie and Li, Wenyang and Liu, Wenbin and Zhan, Bei and Li, Jianfei},
  title = {Large language model-based paper classification framework with key-insight extraction and confidence-weighted voting},
  journal = {Research Synthesis Methods},
  year = {2026},
  doi = {10.1017/rsm.2026.10094}
}
Kostenlos starten →