Validation of large language models (Llama 3 and ChatGPT-4o mini) for title and abstract screening in biomedical systematic reviews
Research Synthesis Methods · 2025-07 · 16 (4) · 620-630
DOI: 10.1017/rsm.2025.15 ↗Kurz gefasst
Validierungsstudie in der Zeitschrift Research Synthesis Methods, online erschienen am 24.03.2025. Verfasst von Adriana López-Pineda und 5 weiteren Autor:innen. Beteiligte Einrichtung: Network for Research on Chronicity, Primary Care and Health Promotion (RICAPPS), San Juan de Alicante, Spain. Hauptthemen laut MEDLINE: Large Language Models, Biomedical Research, Systematic Reviews as Topic. Die Arbeit wurde bisher 21-mal zitiert (Crossref) und nennt 25 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
With the increasing volume of scientific literature, there is a need to streamline the screening process for titles and abstracts in systematic reviews, reduce the workload for reviewers, and minimize errors. This study validated artificial intelligence (AI) tools, specifically Llama 3 70B via Groq’s application programming interface (API) and ChatGPT-4o mini via OpenAI’s API, for automating this process in biomedical research. It compared these AI tools with human reviewers using 1,081 articles after duplicate removal. Each AI model was tested in three configurations to assess sensitivity, specificity, predictive values, and likelihood ratios. The Llama 3 model’s LLA_2 configuration achieved 77.5% sensitivity and 91.4% specificity, with 90.2% accuracy, a positive predictive value (PPV) of 44.3%, and a negative predictive value (NPV) of 97.9%. The ChatGPT-4o mini model’s CHAT_2 configuration showed 56.2% sensitivity, 95.1% specificity, 92.0% accuracy, a PPV of 50.6%, and an NPV of 96.1%. Both models demonstrated strong specificity, with CHAT_2 having higher overall accuracy. Despite these promising results, manual validation remains necessary to address false positives and negatives, ensuring that no important studies are overlooked. This study suggests that AI can significantly enhance efficiency and accuracy in systematic reviews, potentially revolutionizing not only biomedical research but also other fields requiring extensive literature reviews.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Network for Research on Chronicity, Primary Care and Health Promotion (RICAPPS), San Juan de Alicante, Spain.
- Clinical Medicine Department, School of Medicine, Miguel Hernandez University, San Juan de Alicante, Spain.
- Primary Care Research Center, Miguel Hernandez University, San Juan de Alicante, Spain.
- Department of Statistics, Mathematics and Informatics, Miguel Hernandez University, San Juan de Alicante, Spain.
Zitieren
APA
López-Pineda, A., Nouni-García, R., Carbonell-Soliva, Á., Gil-Guillén, V. F., Carratalá-Munuera, C., & Borrás, F. (2025). Validation of large language models (Llama 3 and ChatGPT-4o mini) for title and abstract screening in biomedical systematic reviews. Research Synthesis Methods, 16(4), 620-630. https://doi.org/10.1017/rsm.2025.15
Vancouver
López-Pineda A, Nouni-García R, Carbonell-Soliva Á, Gil-Guillén VF, Carratalá-Munuera C, Borrás F. Validation of large language models (Llama 3 and ChatGPT-4o mini) for title and abstract screening in biomedical systematic reviews. Research Synthesis Methods. 2025;16(4):620-630. doi:10.1017/rsm.2025.15
BibTeX
@article{LpezPineda2025,
author = {López-Pineda, Adriana and Nouni-García, Rauf and Carbonell-Soliva, Álvaro and Gil-Guillén, Vicente F and Carratalá-Munuera, Concepción and Borrás, Fernando},
title = {Validation of large language models (Llama 3 and ChatGPT-4o mini) for title and abstract screening in biomedical systematic reviews},
journal = {Research Synthesis Methods},
year = {2025},
volume = {16},
number = {4},
pages = {620--630},
doi = {10.1017/rsm.2025.15}
}
EUVIMED