EUVIMEDEUVIMED entdecken
ValidierungsstudieOpen Access · CC BY 4.0

Validation of large language models (Llama 3 and ChatGPT-4o mini) for title and abstract screening in biomedical systematic reviews

Adriana López-Pineda, Rauf Nouni-García, Álvaro Carbonell-Soliva, Vicente F Gil-Guillén, Concepción Carratalá-Munuera, Fernando Borrás

Research Synthesis Methods · 2025-07 · 16 (4) · 620-630

DOI: 10.1017/rsm.2025.15 ↗

Kurz gefasst

Validierungsstudie in der Zeitschrift Research Synthesis Methods, online erschienen am 24.03.2025. Verfasst von Adriana López-Pineda und 5 weiteren Autor:innen. Beteiligte Einrichtung: Network for Research on Chronicity, Primary Care and Health Promotion (RICAPPS), San Juan de Alicante, Spain. Hauptthemen laut MEDLINE: Large Language Models, Biomedical Research, Systematic Reviews as Topic. Die Arbeit wurde bisher 21-mal zitiert (Crossref) und nennt 25 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

With the increasing volume of scientific literature, there is a need to streamline the screening process for titles and abstracts in systematic reviews, reduce the workload for reviewers, and minimize errors. This study validated artificial intelligence (AI) tools, specifically Llama 3 70B via Groq’s application programming interface (API) and ChatGPT-4o mini via OpenAI’s API, for automating this process in biomedical research. It compared these AI tools with human reviewers using 1,081 articles after duplicate removal. Each AI model was tested in three configurations to assess sensitivity, specificity, predictive values, and likelihood ratios. The Llama 3 model’s LLA_2 configuration achieved 77.5% sensitivity and 91.4% specificity, with 90.2% accuracy, a positive predictive value (PPV) of 44.3%, and a negative predictive value (NPV) of 97.9%. The ChatGPT-4o mini model’s CHAT_2 configuration showed 56.2% sensitivity, 95.1% specificity, 92.0% accuracy, a PPV of 50.6%, and an NPV of 96.1%. Both models demonstrated strong specificity, with CHAT_2 having higher overall accuracy. Despite these promising results, manual validation remains necessary to address false positives and negatives, ensuring that no important studies are overlooked. This study suggests that AI can significantly enhance efficiency and accuracy in systematic reviews, potentially revolutionizing not only biomedical research but also other fields requiring extensive literature reviews.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsBiomedical ResearchSystematic Reviews as TopicHumansArtificial IntelligenceReproducibility of ResultsSensitivity and SpecificityGenerative Artificial IntelligenceSoftwareAlgorithmsMachine LearningAbstracting and Indexing

Stichwörter der Autor:innen

abstracting and indexingartificial intelligencemachine learningreview literature as a topic

Autor:innen und Institutionen

  • Network for Research on Chronicity, Primary Care and Health Promotion (RICAPPS), San Juan de Alicante, Spain.
  • Clinical Medicine Department, School of Medicine, Miguel Hernandez University, San Juan de Alicante, Spain.
  • Primary Care Research Center, Miguel Hernandez University, San Juan de Alicante, Spain.
  • Department of Statistics, Mathematics and Informatics, Miguel Hernandez University, San Juan de Alicante, Spain.

Zitieren

APA

López-Pineda, A., Nouni-García, R., Carbonell-Soliva, Á., Gil-Guillén, V. F., Carratalá-Munuera, C., & Borrás, F. (2025). Validation of large language models (Llama 3 and ChatGPT-4o mini) for title and abstract screening in biomedical systematic reviews. Research Synthesis Methods, 16(4), 620-630. https://doi.org/10.1017/rsm.2025.15

Vancouver

López-Pineda A, Nouni-García R, Carbonell-Soliva Á, Gil-Guillén VF, Carratalá-Munuera C, Borrás F. Validation of large language models (Llama 3 and ChatGPT-4o mini) for title and abstract screening in biomedical systematic reviews. Research Synthesis Methods. 2025;16(4):620-630. doi:10.1017/rsm.2025.15

BibTeX
@article{LpezPineda2025,
  author = {López-Pineda, Adriana and Nouni-García, Rauf and Carbonell-Soliva, Álvaro and Gil-Guillén, Vicente F and Carratalá-Munuera, Concepción and Borrás, Fernando},
  title = {Validation of large language models (Llama 3 and ChatGPT-4o mini) for title and abstract screening in biomedical systematic reviews},
  journal = {Research Synthesis Methods},
  year = {2025},
  volume = {16},
  number = {4},
  pages = {620--630},
  doi = {10.1017/rsm.2025.15}
}
Kostenlos starten →