Zero-shot interpretable biomedical literature appraisal with generative large language models
JAMIA Open · 06.03.2026 · 9 (2)
DOI: 10.1093/jamiaopen/ooag043 ↗Kurz gefasst
Fachartikel in der Zeitschrift JAMIA Open, online erschienen am 09.04.2026. Verfasst von Fangwen Zhou und 6 weiteren Autor:innen. Beteiligte Einrichtung: Health Information Research Unit, Department of Health Research Methods, Evidence, and Impact, Faculty of Health Sciences, McMaster University, Hamil…. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 53 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: Mitacs Business Strategy Internship.
Abstract (Originalsprache)
Objective
This study aims to apply 2 decoder-based Generative Pre-trained Transformer (GPT) models (GPT-4o and GPT-o3-mini) in automating the methodological appraisal of randomized controlled trials (RCTs), under a variety of prompt designs, and to compare their performance to a fine-tuned encoder-only BioLinkBERT model Materials and Methods A stratified random sample of 800 articles from the McMaster Premium LiteratUre Service and Clinical Hedges databases was appraised using 2 prompting schemes: (1) classifier (independent assessment) and (2) verifier (validation of BioLinkBERT) considering either the title and abstract (TIAB) or the full text of an article. Performance was primarily evaluated against human assessments using Matthews correlation coefficient (MCC). Bootstrapping over 1000 iterations was used to estimate 95% CIs.
Results
GPT-4o as a classifier with full text demonstrated comparable performance (MCC 0.429; 95% CI, 0.387-0.470) to BioLinkBERT (MCC, 0.466; 95% CI, 0.409-0.519), drastically outperforming the best GPT-o3-mini scheme (MCC, 0.272; 95% CI, 0.211-0.334). GPT-4o as a verifier with full text showed similar performance (MCC, 0.391; 95% CI, 0.335-0.444). GPT models provided transparent criterion-specific justifications. Performance using TIAB alone markedly decreased for GPT models (MCC, ≤0.100), highlighting dependency on detailed methodological information.
Discussion
GPT-4o effectively automates RCT critical appraisal with comparable performance to specialized fine-tuned models when provided full text, enhancing interpretability and transparency through explicit justifications. Limitations in abstract-level detail suggest complementary roles for fine-tuned models when full texts are unavailable. Future studies should optimize goal-specific prompting to further facilitate adoption in clinical knowledge translation workflows.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Health Information Research Unit, Department of Health Research Methods, Evidence, and Impact, Faculty of Health Sciences, McMaster University, Hamilton, ON L8S 4K1, Canada.
- Department of Computer Science, Birmingham City University, Birmingham, B4 7BD, United Kingdom.
- Department of Oncology, Faculty of Health Sciences, McMaster University, Hamilton, ON L8V 5C2, Canada.
- Department of Medicine, Faculty of Health Sciences, McMaster University, Hamilton, ON L8S 4K1, Canada.
ORCID-Profile: Alfonso Iorio, Cynthia Lokker
Förderung
- Mitacs Business Strategy Internship
Zitieren
APA
Zhou, F., Afzal, M., Saha, A., Parrish, R., Haynes, R. B., Iorio, A., & Lokker, C. (2026). Zero-shot interpretable biomedical literature appraisal with generative large language models. JAMIA Open, 9(2). https://doi.org/10.1093/jamiaopen/ooag043
Vancouver
Zhou F, Afzal M, Saha A, Parrish R, Haynes RB, Iorio A, et al. Zero-shot interpretable biomedical literature appraisal with generative large language models. JAMIA Open. 2026;9(2). doi:10.1093/jamiaopen/ooag043
BibTeX
@article{Zhou2026,
author = {Zhou, Fangwen and Afzal, Muhammad and Saha, Ashirbani and Parrish, Rick and Haynes, R Brian and Iorio, Alfonso and Lokker, Cynthia},
title = {Zero-shot interpretable biomedical literature appraisal with generative large language models},
journal = {JAMIA Open},
year = {2026},
volume = {9},
number = {2},
doi = {10.1093/jamiaopen/ooag043}
}
EUVIMED