EUVIMEDEUVIMED entdecken
Open Access

Collaborative large language models for automated data extraction in living systematic reviews

Muhammad Ali Khan, Umair Ayub, Syed Arsalan Ahmed Naqvi, Kaneez Zahra Rubab Khakwani, Zaryab Bin Riaz Sipra, Ammad Raina, Sihan Zhou, Huan He iD, Amir Saeidi, Bashar Hasan, Robert Bryan Rumble, Danielle S Bitterman und 9 weitere

Journal of the American Medical Informatics Association · 01.04.2025 · 32 (4) · 638-647

DOI: 10.1093/jamia/ocae325 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 21.01.2025. Verfasst von Muhammad Ali Khan und 20 weiteren Autor:innen. Beteiligte Einrichtung: Department of Medicine, Mayo Clinic, Phoenix, AZ, 85054, United States. Hauptthemen laut MEDLINE: Systematic Reviews as Topic, Information Storage and Retrieval, Natural Language Processing, Data Mining. Die Arbeit wurde bisher 49-mal zitiert (Crossref) und nennt 60 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NCI NIH HHS, NIH HHS, AstraZeneca.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

Data extraction from the published literature is the most laborious step in conducting living systematic reviews (LSRs). We aim to build a generalizable, automated data extraction workflow leveraging large language models (LLMs) that mimics the real-world 2-reviewer process. Materials and Methods A dataset of 10 trials (22 publications) from a published LSR was used, focusing on 23 variables related to trial, population, and outcomes data. The dataset was split into prompt development (n = 5) and held-out test sets (n = 17). GPT-4-turbo and Claude-3-Opus were used for data extraction. Responses from the 2 LLMs were considered concordant if they were the same for a given variable. The discordant responses from each LLM were provided to the other LLM for cross-critique. Accuracy, ie, the total number of correct responses divided by the total number of responses, was computed to assess performance.

Results

In the prompt development set, 110 (96%) responses were concordant, achieving an accuracy of 0.99 against the gold standard. In the test set, 342 (87%) responses were concordant. The accuracy of the concordant responses was 0.94. The accuracy of the discordant responses was 0.41 for GPT-4-turbo and 0.50 for Claude-3-Opus. Of the 49 discordant responses, 25 (51%) became concordant after cross-critique, increasing accuracy to 0.76.

Discussion

Concordant responses by the LLMs are likely to be accurate. In instances of discordant responses, cross-critique can further increase the accuracy.

Conclusion

Large language models, when simulated in a collaborative, 2-reviewer workflow, can extract data with reasonable performance, enabling truly “living” systematic reviews.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Systematic Reviews as TopicInformation Storage and RetrievalNatural Language ProcessingData MiningHumansWorkflowLarge Language Models

Stichwörter der Autor:innen

data extractionlarge language modelsmeta-analysisnatural language processingsystematic review

Autor:innen und Institutionen

  • Department of Medicine, Mayo Clinic, Phoenix, AZ, 85054, United States.
  • Department of Medicine, University of Arizona, Tucson, AZ, 85721, United States.
  • Department of Medicine and Surgery, Rashid Latif Medical College, Lahore, Punjab, 54000, Pakistan.
  • Department of Medicine, Canyon Vista Hospital, Sierra Vista, AZ, 85635, United States.
  • Department of Biomedical Informatics and Data Science, Yale University, New Haven, CT, 06520, United States.
  • Department of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ, 85287, United States.

ORCID-Profile: Huan He

Förderung

  • NCI NIH HHS
  • NIH HHS
  • AstraZeneca
  • National Institutes of Health

Zitieren

APA

Khan, M. A., Ayub, U., Naqvi, S. A. A., Khakwani, K. Z. R., Sipra, Z. B. R., Raina, A., Zhou, S., He, H., Saeidi, A., Hasan, B., Rumble, R. B., Bitterman, D. S., Warner, J. L., Zou, J., Tevaarwerk, A. J., Leventakos, K., Kehl, K. L., Palmer, J. M., Murad, M. H., …, & Riaz, I. B. (2025). Collaborative large language models for automated data extraction in living systematic reviews. Journal of the American Medical Informatics Association, 32(4), 638-647. https://doi.org/10.1093/jamia/ocae325

Vancouver

Khan MA, Ayub U, Naqvi SAA, Khakwani KZR, Sipra ZBR, Raina A, et al. Collaborative large language models for automated data extraction in living systematic reviews. Journal of the American Medical Informatics Association. 2025;32(4):638-647. doi:10.1093/jamia/ocae325

BibTeX
@article{Khan2025,
  author = {Khan, Muhammad Ali and Ayub, Umair and Naqvi, Syed Arsalan Ahmed and Khakwani, Kaneez Zahra Rubab and Sipra, Zaryab Bin Riaz and Raina, Ammad and Zhou, Sihan and He, Huan and Saeidi, Amir and Hasan, Bashar and Rumble, Robert Bryan and Bitterman, Danielle S and Warner, Jeremy L and Zou, Jia and Tevaarwerk, Amye J and Leventakos, Konstantinos and Kehl, Kenneth L and Palmer, Jeanne M and Murad, Mohammad Hassan and Baral, Chitta and Riaz, Irbaz Bin},
  title = {Collaborative large language models for automated data extraction in living systematic reviews},
  journal = {Journal of the American Medical Informatics Association},
  year = {2025},
  volume = {32},
  number = {4},
  pages = {638--647},
  doi = {10.1093/jamia/ocae325}
}
Kostenlos starten →