EUVIMEDEUVIMED entdecken
Open Access

Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study

Eddie Guo iD, Mehul Gupta iD, Jiawen Deng iD, Ye-Jean Park iD, Michael Paget iD, Christopher Naugler iD

Journal of Medical Internet Research · 12.01.2024 · 26 · e48996

DOI: 10.2196/48996 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Medical Internet Research, online erschienen am 12.01.2024. Verfasst von Eddie Guo und 5 weiteren Autor:innen. Beteiligte Einrichtung: Cumming School of Medicine, University of Calgary, Calgary, AB, Canada. Hauptthemen laut MEDLINE: Biomedical Research, Systematic Reviews as Topic, Artificial Intelligence. Die Arbeit wurde bisher 215-mal zitiert (Crossref) und nennt 39 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

The systematic review of clinical research papers is a labor-intensive and time-consuming process that often involves the screening of thousands of titles and abstracts. The accuracy and efficiency of this process are critical for the quality of the review and subsequent health care decisions. Traditional methods rely heavily on human reviewers, often requiring a significant investment of time and resources.

Objective

This study aims to assess the performance of the OpenAI generative pretrained transformer (GPT) and GPT-4 application programming interfaces (APIs) in accurately and efficiently identifying relevant titles and abstracts from real-world clinical review data sets and comparing their performance against ground truth labeling by 2 independent human reviewers.

Methods

We introduce a novel workflow using the Chat GPT and GPT-4 APIs for screening titles and abstracts in clinical reviews. A Python script was created to make calls to the API with the screening criteria in natural language and a corpus of title and abstract data sets filtered by a minimum of 2 human reviewers. We compared the performance of our model against human-reviewed papers across 6 review papers, screening over 24,000 titles and abstracts.

Results

Our results show an accuracy of 0.91, a macro F1-score of 0.60, a sensitivity of excluded papers of 0.91, and a sensitivity of included papers of 0.76. The interrater variability between 2 independent human screeners was κ=0.46, and the prevalence and bias-adjusted κ between our proposed methods and the consensus-based human decisions was κ=0.96. On a randomly selected subset of papers, the GPT models demonstrated the ability to provide reasoning for their decisions and corrected their initial decisions upon being asked to explain their reasoning for incorrect classifications.

Conclusions

Large language models have the potential to streamline the clinical review process, save valuable time and effort for researchers, and contribute to the overall quality of clinical reviews. By prioritizing the workflow and acting as an aid rather than a replacement for researchers and reviewers, models such as GPT-4 can enhance efficiency and lead to more accurate and reliable conclusions in medical research.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Biomedical ResearchSystematic Reviews as TopicArtificial IntelligenceHumansConsensusData AnalysisProblem SolvingNatural Language ProcessingWorkflow

Stichwörter der Autor:innen

Chat GPTGPTGPT-4LLMNLPabstract screeningclassificationextractextractionfree textlanguage modellarge language modelsnatural language processingnonopiod analgesiareview methodology

Autor:innen und Institutionen

  • Cumming School of Medicine, University of Calgary, Calgary, AB, Canada.
  • Temerty Faculty of Medicine, University of Toronto, Toronto, AB, Canada.

ORCID-Profile: Eddie Guo, Mehul Gupta, Jiawen Deng, Ye-Jean Park, Michael Paget, Christopher Naugler

Zitieren

APA

Guo, E., Gupta, M., Deng, J., Park, Y. J., Paget, M., & Naugler, C. (2024). Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study. Journal of Medical Internet Research, 26, e48996. https://doi.org/10.2196/48996

Vancouver

Guo E, Gupta M, Deng J, Park YJ, Paget M, Naugler C. Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study. Journal of Medical Internet Research. 2024;26:e48996. doi:10.2196/48996

BibTeX
@article{Guo2024,
  author = {Guo, Eddie and Gupta, Mehul and Deng, Jiawen and Park, Ye-Jean and Paget, Michael and Naugler, Christopher},
  title = {Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  volume = {26},
  pages = {e48996},
  doi = {10.2196/48996}
}
Kostenlos starten →