EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Can large language models replace humans in systematic reviews? Evaluating GPT ‐4's efficacy in screening and extracting data from peer‐reviewed and grey literature in multiple languages

Qusai Khraisha iD, Sophie Put, Johanna Kappenberg, Azza Warraitch, Kristin Hadfield iD

Research Synthesis Methods · 2024-07 · 15 (4) · 616-626

DOI: 10.1002/jrsm.1715 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Research Synthesis Methods, online erschienen am 14.03.2024. Verfasst von Qusai Khraisha und 4 weiteren Autor:innen. Beteiligte Einrichtung: Trinity Centre for Global Health, Trinity College Dublin, Dublin, Ireland. Hauptthemen laut MEDLINE: Systematic Reviews as Topic, Artificial Intelligence. Die Arbeit wurde bisher 303-mal zitiert (Crossref) und nennt 42 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Systematic reviews are vital for guiding practice, research and policy, although they are often slow and labour‐intensive. Large language models (LLMs) could speed up and automate systematic reviews, but their performance in such tasks has yet to be comprehensively evaluated against humans, and no study has tested Generative Pre‐Trained Transformer (GPT)‐4, the biggest LLM so far. This pre‐registered study uses a “human‐out‐of‐the‐loop” approach to evaluate GPT‐4's capability in title/abstract screening, full‐text review and data extraction across various literature types and languages. Although GPT‐4 had accuracy on par with human performance in some tasks, results were skewed by chance agreement and dataset imbalance. Adjusting for these caused performance scores to drop across all stages: for data extraction, performance was moderate, and for screening, it ranged from none in highly balanced literature datasets (~1:1) to moderate in those datasets where the ratio of inclusion to exclusion in studies was imbalanced (~1:3). When screening full‐text literature using highly reliable prompts, GPT‐4's performance was more robust, reaching “human‐like” levels. Although our findings indicate that, currently, substantial caution should be exercised if LLMs are being used to conduct systematic reviews, they also offer preliminary evidence that, for certain review tasks delivered under specific conditions, LLMs can rival human performance.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Systematic Reviews as TopicArtificial IntelligenceAlgorithmsInformation Storage and RetrievalReproducibility of ResultsMachine LearningNatural Language Processing

Stichwörter der Autor:innen

GPTartificial intelligence (AI)large language models (LLMs)machine learningnatural language processing (NLP)systematic reviews

Autor:innen und Institutionen

  • Trinity Centre for Global Health, Trinity College Dublin, Dublin, Ireland.
  • School of Psychology, Trinity College Dublin, Dublin, Ireland.
  • Department of Education, York University, York, UK.

ORCID-Profile: Qusai Khraisha, Kristin Hadfield

Zitieren

APA

Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2024). Can large language models replace humans in systematic reviews? Evaluating GPT ‐4's efficacy in screening and extracting data from peer‐reviewed and grey literature in multiple languages. Research Synthesis Methods, 15(4), 616-626. https://doi.org/10.1002/jrsm.1715

Vancouver

Khraisha Q, Put S, Kappenberg J, Warraitch A, Hadfield K. Can large language models replace humans in systematic reviews? Evaluating GPT ‐4's efficacy in screening and extracting data from peer‐reviewed and grey literature in multiple languages. Research Synthesis Methods. 2024;15(4):616-626. doi:10.1002/jrsm.1715

BibTeX
@article{Khraisha2024,
  author = {Khraisha, Qusai and Put, Sophie and Kappenberg, Johanna and Warraitch, Azza and Hadfield, Kristin},
  title = {Can large language models replace humans in systematic reviews? Evaluating GPT ‐4's efficacy in screening and extracting data from peer‐reviewed and grey literature in multiple languages},
  journal = {Research Synthesis Methods},
  year = {2024},
  volume = {15},
  number = {4},
  pages = {616--626},
  doi = {10.1002/jrsm.1715}
}
Kostenlos starten →