EUVIMEDEUVIMED entdecken
ValidierungsstudieSystematische ÜbersichtOpen Access · CC BY-NC 4.0

Large Language Model-Assisted Systematic Review: Validation Based on Cochrane Review Data

Siun Kim, Hyung-Jin Yoon

Studies in Health Technology and Informatics · 2025

DOI: 10.3233/shti250501 ↗

Kurz gefasst

Validierungsstudie in der Zeitschrift Studies in Health Technology and Informatics, 2025. Verfasst von Siun Kim und 1 weiteren Autor:innen. Beteiligte Einrichtung: Biomedical Research Institute, Seoul National University Hospital, Seoul, Korea. Hauptthemen laut MEDLINE: Natural Language Processing, Systematic Reviews as Topic, Evidence-Based Medicine, Programming Languages. Die Arbeit wurde bisher 1-mal zitiert (Crossref). Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large Language Models (LLMs) offer potential for automating systematic reviews, a labor-intensive process in evidence-based medicine. We evaluated GPT-4o, GPT-4o-mini, and Llama 3.1:8B on abstract screening and risk of bias assessment using 12 Cochrane drug intervention reviews. GPT-4o achieved the best screening performance (recall 0.894, precision 0.492). We propose a one-shot inclusivity adjustment method enabling threshold modulation without repeated inferences. For risk of bias, accuracy varied by domain, highest in random sequence generation (0.873), and lowest in selective reporting (0.418). Our findings demonstrate LLMs’ practical utility and current limitations in automating systematic reviews.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language ProcessingSystematic Reviews as TopicEvidence-Based MedicineProgramming LanguagesReview Literature as TopicReproducibility of ResultsHumansLarge Language Models

Stichwörter der Autor:innen

Abstract ScreeningLarge Language ModelsRisk of BiasSystematic Review

Autor:innen und Institutionen

  • Biomedical Research Institute, Seoul National University Hospital, Seoul, Korea.
  • Department of Biomedical Engineering, Seoul National University, Seoul, Korea.
  • Medical Bigdata Research Center, Seoul National University, Seoul, Korea.

Zitieren

APA

Kim, S., & Yoon, H. J. (2025). Large Language Model-Assisted Systematic Review: Validation Based on Cochrane Review Data. Studies in Health Technology and Informatics. https://doi.org/10.3233/shti250501

Vancouver

Kim S, Yoon HJ. Large Language Model-Assisted Systematic Review: Validation Based on Cochrane Review Data. Studies in Health Technology and Informatics. 2025. doi:10.3233/shti250501

BibTeX
@article{Kim2025,
  author = {Kim, Siun and Yoon, Hyung-Jin},
  title = {Large Language Model-Assisted Systematic Review: Validation Based on Cochrane Review Data},
  journal = {Studies in Health Technology and Informatics},
  year = {2025},
  doi = {10.3233/shti250501}
}
Kostenlos starten →