Collaborative large language models for automated data extraction in living systematic reviews
Journal of the American Medical Informatics Association · 01.04.2025 · 32 (4) · 638-647
DOI: 10.1093/jamia/ocae325 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 21.01.2025. Verfasst von Muhammad Ali Khan und 20 weiteren Autor:innen. Beteiligte Einrichtung: Department of Medicine, Mayo Clinic, Phoenix, AZ, 85054, United States. Hauptthemen laut MEDLINE: Systematic Reviews as Topic, Information Storage and Retrieval, Natural Language Processing, Data Mining. Die Arbeit wurde bisher 49-mal zitiert (Crossref) und nennt 60 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NCI NIH HHS, NIH HHS, AstraZeneca.
Abstract (Originalsprache)
Objective
Data extraction from the published literature is the most laborious step in conducting living systematic reviews (LSRs). We aim to build a generalizable, automated data extraction workflow leveraging large language models (LLMs) that mimics the real-world 2-reviewer process. Materials and Methods A dataset of 10 trials (22 publications) from a published LSR was used, focusing on 23 variables related to trial, population, and outcomes data. The dataset was split into prompt development (n = 5) and held-out test sets (n = 17). GPT-4-turbo and Claude-3-Opus were used for data extraction. Responses from the 2 LLMs were considered concordant if they were the same for a given variable. The discordant responses from each LLM were provided to the other LLM for cross-critique. Accuracy, ie, the total number of correct responses divided by the total number of responses, was computed to assess performance.
Results
In the prompt development set, 110 (96%) responses were concordant, achieving an accuracy of 0.99 against the gold standard. In the test set, 342 (87%) responses were concordant. The accuracy of the concordant responses was 0.94. The accuracy of the discordant responses was 0.41 for GPT-4-turbo and 0.50 for Claude-3-Opus. Of the 49 discordant responses, 25 (51%) became concordant after cross-critique, increasing accuracy to 0.76.
Discussion
Concordant responses by the LLMs are likely to be accurate. In instances of discordant responses, cross-critique can further increase the accuracy.
Conclusion
Large language models, when simulated in a collaborative, 2-reviewer workflow, can extract data with reasonable performance, enabling truly “living” systematic reviews.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Medicine, Mayo Clinic, Phoenix, AZ, 85054, United States.
- Department of Medicine, University of Arizona, Tucson, AZ, 85721, United States.
- Department of Medicine and Surgery, Rashid Latif Medical College, Lahore, Punjab, 54000, Pakistan.
- Department of Medicine, Canyon Vista Hospital, Sierra Vista, AZ, 85635, United States.
- Department of Biomedical Informatics and Data Science, Yale University, New Haven, CT, 06520, United States.
- Department of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ, 85287, United States.
ORCID-Profile: Huan He
Förderung
- NCI NIH HHS
- NIH HHS
- AstraZeneca
- National Institutes of Health
Zitieren
APA
Khan, M. A., Ayub, U., Naqvi, S. A. A., Khakwani, K. Z. R., Sipra, Z. B. R., Raina, A., Zhou, S., He, H., Saeidi, A., Hasan, B., Rumble, R. B., Bitterman, D. S., Warner, J. L., Zou, J., Tevaarwerk, A. J., Leventakos, K., Kehl, K. L., Palmer, J. M., Murad, M. H., …, & Riaz, I. B. (2025). Collaborative large language models for automated data extraction in living systematic reviews. Journal of the American Medical Informatics Association, 32(4), 638-647. https://doi.org/10.1093/jamia/ocae325
Vancouver
Khan MA, Ayub U, Naqvi SAA, Khakwani KZR, Sipra ZBR, Raina A, et al. Collaborative large language models for automated data extraction in living systematic reviews. Journal of the American Medical Informatics Association. 2025;32(4):638-647. doi:10.1093/jamia/ocae325
BibTeX
@article{Khan2025,
author = {Khan, Muhammad Ali and Ayub, Umair and Naqvi, Syed Arsalan Ahmed and Khakwani, Kaneez Zahra Rubab and Sipra, Zaryab Bin Riaz and Raina, Ammad and Zhou, Sihan and He, Huan and Saeidi, Amir and Hasan, Bashar and Rumble, Robert Bryan and Bitterman, Danielle S and Warner, Jeremy L and Zou, Jia and Tevaarwerk, Amye J and Leventakos, Konstantinos and Kehl, Kenneth L and Palmer, Jeanne M and Murad, Mohammad Hassan and Baral, Chitta and Riaz, Irbaz Bin},
title = {Collaborative large language models for automated data extraction in living systematic reviews},
journal = {Journal of the American Medical Informatics Association},
year = {2025},
volume = {32},
number = {4},
pages = {638--647},
doi = {10.1093/jamia/ocae325}
}
EUVIMED