EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Zero-shot interpretable biomedical literature appraisal with generative large language models

Fangwen Zhou, Muhammad Afzal, Ashirbani Saha, Rick Parrish, R Brian Haynes, Alfonso Iorio iD, Cynthia Lokker iD

JAMIA Open · 06.03.2026 · 9 (2)

DOI: 10.1093/jamiaopen/ooag043 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JAMIA Open, online erschienen am 09.04.2026. Verfasst von Fangwen Zhou und 6 weiteren Autor:innen. Beteiligte Einrichtung: Health Information Research Unit, Department of Health Research Methods, Evidence, and Impact, Faculty of Health Sciences, McMaster University, Hamil…. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 53 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: Mitacs Business Strategy Internship.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

This study aims to apply 2 decoder-based Generative Pre-trained Transformer (GPT) models (GPT-4o and GPT-o3-mini) in automating the methodological appraisal of randomized controlled trials (RCTs), under a variety of prompt designs, and to compare their performance to a fine-tuned encoder-only BioLinkBERT model Materials and Methods A stratified random sample of 800 articles from the McMaster Premium LiteratUre Service and Clinical Hedges databases was appraised using 2 prompting schemes: (1) classifier (independent assessment) and (2) verifier (validation of BioLinkBERT) considering either the title and abstract (TIAB) or the full text of an article. Performance was primarily evaluated against human assessments using Matthews correlation coefficient (MCC). Bootstrapping over 1000 iterations was used to estimate 95% CIs.

Results

GPT-4o as a classifier with full text demonstrated comparable performance (MCC 0.429; 95% CI, 0.387-0.470) to BioLinkBERT (MCC, 0.466; 95% CI, 0.409-0.519), drastically outperforming the best GPT-o3-mini scheme (MCC, 0.272; 95% CI, 0.211-0.334). GPT-4o as a verifier with full text showed similar performance (MCC, 0.391; 95% CI, 0.335-0.444). GPT models provided transparent criterion-specific justifications. Performance using TIAB alone markedly decreased for GPT models (MCC, ≤0.100), highlighting dependency on detailed methodological information.

Discussion

GPT-4o effectively automates RCT critical appraisal with comparable performance to specialized fine-tuned models when provided full text, enhancing interpretability and transparency through explicit justifications. Limitations in abstract-level detail suggest complementary roles for fine-tuned models when full texts are unavailable. Future studies should optimize goal-specific prompting to further facilitate adoption in clinical knowledge translation workflows.

Schlagwörter

Stichwörter der Autor:innen

GPTdeep learningevidence-based medicineexplainable AInatural language processingtext classification

Autor:innen und Institutionen

  • Health Information Research Unit, Department of Health Research Methods, Evidence, and Impact, Faculty of Health Sciences, McMaster University, Hamilton, ON L8S 4K1, Canada.
  • Department of Computer Science, Birmingham City University, Birmingham, B4 7BD, United Kingdom.
  • Department of Oncology, Faculty of Health Sciences, McMaster University, Hamilton, ON L8V 5C2, Canada.
  • Department of Medicine, Faculty of Health Sciences, McMaster University, Hamilton, ON L8S 4K1, Canada.

ORCID-Profile: Alfonso Iorio, Cynthia Lokker

Förderung

  • Mitacs Business Strategy Internship

Zitieren

APA

Zhou, F., Afzal, M., Saha, A., Parrish, R., Haynes, R. B., Iorio, A., & Lokker, C. (2026). Zero-shot interpretable biomedical literature appraisal with generative large language models. JAMIA Open, 9(2). https://doi.org/10.1093/jamiaopen/ooag043

Vancouver

Zhou F, Afzal M, Saha A, Parrish R, Haynes RB, Iorio A, et al. Zero-shot interpretable biomedical literature appraisal with generative large language models. JAMIA Open. 2026;9(2). doi:10.1093/jamiaopen/ooag043

BibTeX
@article{Zhou2026,
  author = {Zhou, Fangwen and Afzal, Muhammad and Saha, Ashirbani and Parrish, Rick and Haynes, R Brian and Iorio, Alfonso and Lokker, Cynthia},
  title = {Zero-shot interpretable biomedical literature appraisal with generative large language models},
  journal = {JAMIA Open},
  year = {2026},
  volume = {9},
  number = {2},
  doi = {10.1093/jamiaopen/ooag043}
}
Kostenlos starten →