EUVIMEDEUVIMED entdecken
Open Access

Enabling inclusive systematic reviews: incorporating preprint articles with large language model-driven evaluations

Rui Yang iD, Jiayi Tong, Haoyuan Wang, Hui Huang, Ziyang Hu, Peiyu Li, Nan Liu, Christopher J Lindsell, Michael J Pencina iD, Yong Chen iD, Chuan Hong iD

Journal of the American Medical Informatics Association · 2025

DOI: 10.1093/jamia/ocaf137 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, 2025. Verfasst von Rui Yang und 10 weiteren Autor:innen. Beteiligte Einrichtung: Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore, 169857, Singapore. Hauptthemen laut MEDLINE: Comparative Effectiveness Research, Natural Language Processing, Publishing, Review Literature as Topic. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 33 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIH HHS, NIA NIH HHS, NLM NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

Systematic reviews in comparative effectiveness research require timely evidence synthesis. With the rapid advancement of medical research, preprint articles play an increasingly important role in accelerating knowledge dissemination. However, as preprint articles are not peer-reviewed before publication, their quality varies significantly, posing challenges for evidence inclusion in systematic reviews. Materials and Methods We developed AutoConfidenceScore (automated confidence score assessment), an advanced framework for predicting preprint publication, which reduces reliance on manual curation and expands the range of predictors, including three key advancements: (1) automated data extraction using natural language processing techniques, (2) semantic embeddings of titles and abstracts, and (3) large language model (LLM)-driven evaluation scores. Additionally, we employed two prediction models: a random forest classifier for binary outcome and a survival cure model that predicts both binary outcome and publication risk over time.

Results

The random forest classifier achieved an area under the receiver operating characteristic curve (AUROC) of 0.747 using all features. The survival cure model achieved an AUROC of 0.731 for binary outcome prediction and a concordance index of 0.667 for time-to-publication risk.

Discussion

Our study advances the framework for preprint publication prediction through automated data extraction and multiple feature integration. By combining semantic embeddings with LLM-driven evaluations, AutoConfidenceScore significantly enhances predictive performance while reducing manual annotation burden.

Conclusion

AutoConfidenceScore has the potential to facilitate incorporation of preprint articles during the appraisal phase of systematic reviews, supporting researchers in more effective utilization of preprint resources.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Comparative Effectiveness ResearchNatural Language ProcessingPublishingReview Literature as TopicSystematic Reviews as TopicInformation Storage and RetrievalLarge Language ModelsROC CurveSemantics

Stichwörter der Autor:innen

evidence synthesisevidence-based medicinelarge language modelspreprint articlesystematic reviews

Autor:innen und Institutionen

  • Centre for Quantitative Medicine, Duke-NUS Medical School, Singapore, 169857, Singapore.
  • Department of Biostatistics, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, 21205, United States.
  • Department of Biostatistics and Bioinformatics, Duke School of Medicine, Durham, NC, 27710, United States.
  • Children's Healthcare of Atlanta, Emory University School of Medicine, Atlanta, GA, 30329, United States.
  • Department of Genome Sciences, University of Virginia, Charlottesville, VA, 22903, United States.
  • Duke Clinical Research Institute, Duke University, Durham, NC, 27701, United States.

ORCID-Profile: Rui Yang, Michael J Pencina, Yong Chen, Chuan Hong

Förderung

  • NIH HHS
  • NIA NIH HHS
  • NLM NIH HHS
  • NIMH NIH HHS
  • NIAID NIH HHS
  • NCATS NIH HHS
  • NIDDK NIH HHS
  • NEI NIH HHS

Zitieren

APA

Yang, R., Tong, J., Wang, H., Huang, H., Hu, Z., Li, P., Liu, N., Lindsell, C. J., Pencina, M. J., Chen, Y., & Hong, C. (2025). Enabling inclusive systematic reviews: incorporating preprint articles with large language model-driven evaluations. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocaf137

Vancouver

Yang R, Tong J, Wang H, Huang H, Hu Z, Li P, et al. Enabling inclusive systematic reviews: incorporating preprint articles with large language model-driven evaluations. Journal of the American Medical Informatics Association. 2025. doi:10.1093/jamia/ocaf137

BibTeX
@article{Yang2025,
  author = {Yang, Rui and Tong, Jiayi and Wang, Haoyuan and Huang, Hui and Hu, Ziyang and Li, Peiyu and Liu, Nan and Lindsell, Christopher J and Pencina, Michael J and Chen, Yong and Hong, Chuan},
  title = {Enabling inclusive systematic reviews: incorporating preprint articles with large language model-driven evaluations},
  journal = {Journal of the American Medical Informatics Association},
  year = {2025},
  doi = {10.1093/jamia/ocaf137}
}
Kostenlos starten →