Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study
Journal of Medical Internet Research · 12.01.2024 · 26 · e48996
DOI: 10.2196/48996 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Medical Internet Research, online erschienen am 12.01.2024. Verfasst von Eddie Guo und 5 weiteren Autor:innen. Beteiligte Einrichtung: Cumming School of Medicine, University of Calgary, Calgary, AB, Canada. Hauptthemen laut MEDLINE: Biomedical Research, Systematic Reviews as Topic, Artificial Intelligence. Die Arbeit wurde bisher 215-mal zitiert (Crossref) und nennt 39 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
The systematic review of clinical research papers is a labor-intensive and time-consuming process that often involves the screening of thousands of titles and abstracts. The accuracy and efficiency of this process are critical for the quality of the review and subsequent health care decisions. Traditional methods rely heavily on human reviewers, often requiring a significant investment of time and resources.
Objective
This study aims to assess the performance of the OpenAI generative pretrained transformer (GPT) and GPT-4 application programming interfaces (APIs) in accurately and efficiently identifying relevant titles and abstracts from real-world clinical review data sets and comparing their performance against ground truth labeling by 2 independent human reviewers.
Methods
We introduce a novel workflow using the Chat GPT and GPT-4 APIs for screening titles and abstracts in clinical reviews. A Python script was created to make calls to the API with the screening criteria in natural language and a corpus of title and abstract data sets filtered by a minimum of 2 human reviewers. We compared the performance of our model against human-reviewed papers across 6 review papers, screening over 24,000 titles and abstracts.
Results
Our results show an accuracy of 0.91, a macro F1-score of 0.60, a sensitivity of excluded papers of 0.91, and a sensitivity of included papers of 0.76. The interrater variability between 2 independent human screeners was κ=0.46, and the prevalence and bias-adjusted κ between our proposed methods and the consensus-based human decisions was κ=0.96. On a randomly selected subset of papers, the GPT models demonstrated the ability to provide reasoning for their decisions and corrected their initial decisions upon being asked to explain their reasoning for incorrect classifications.
Conclusions
Large language models have the potential to streamline the clinical review process, save valuable time and effort for researchers, and contribute to the overall quality of clinical reviews. By prioritizing the workflow and acting as an aid rather than a replacement for researchers and reviewers, models such as GPT-4 can enhance efficiency and lead to more accurate and reliable conclusions in medical research.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Cumming School of Medicine, University of Calgary, Calgary, AB, Canada.
- Temerty Faculty of Medicine, University of Toronto, Toronto, AB, Canada.
ORCID-Profile: Eddie Guo, Mehul Gupta, Jiawen Deng, Ye-Jean Park, Michael Paget, Christopher Naugler
Zitieren
APA
Guo, E., Gupta, M., Deng, J., Park, Y. J., Paget, M., & Naugler, C. (2024). Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study. Journal of Medical Internet Research, 26, e48996. https://doi.org/10.2196/48996
Vancouver
Guo E, Gupta M, Deng J, Park YJ, Paget M, Naugler C. Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study. Journal of Medical Internet Research. 2024;26:e48996. doi:10.2196/48996
BibTeX
@article{Guo2024,
author = {Guo, Eddie and Gupta, Mehul and Deng, Jiawen and Park, Ye-Jean and Paget, Michael and Naugler, Christopher},
title = {Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study},
journal = {Journal of Medical Internet Research},
year = {2024},
volume = {26},
pages = {e48996},
doi = {10.2196/48996}
}
EUVIMED