Advancing Accuracy in Multimodal Medical Tasks Through Bootstrapped Language-Image Pretraining (BioMedBLIP): Performance Evaluation Study
JMIR Medical Informatics · 2024-08-05 · 12 · e56627
DOI: 10.2196/56627 ↗At a glance
Journal article in JMIR Medical Informatics, published online on 2024-08-05. Written by Usman Naseem and 2 co-authors. Institution: School of Computing, Macquarie University, Sydney, Australia. Cited 8 times so far (Crossref); 35 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
Medical image analysis, particularly in the context of visual question answering (VQA) and image captioning, is crucial for accurate diagnosis and educational purposes.
Objective
Our study aims to introduce BioMedBLIP models, fine-tuned for VQA tasks using specialized medical data sets such as Radiology Objects in Context and Medical Information Mart for Intensive Care-Chest X-ray, and evaluate their performance in comparison to the state of the art (SOTA) original Bootstrapping Language-Image Pretraining (BLIP) model.
Methods
We present 9 versions of BioMedBLIP across 3 downstream tasks in various data sets. The models are trained on a varying number of epochs. The findings indicate the strong overall performance of our models. We proposed BioMedBLIP for the VQA generation model, VQA classification model, and BioMedBLIP image caption model. We conducted pretraining in BLIP using medical data sets, producing an adapted BLIP model tailored for medical applications.
Results
In VQA generation tasks, BioMedBLIP models outperformed the SOTA on the Semantically-Labeled Knowledge-Enhanced (SLAKE) data set, VQA in Radiology (VQA-RAD), and Image Cross-Language Evaluation Forum data sets. In VQA classification, our models consistently surpassed the SOTA on the SLAKE data set. Our models also showed competitive performance on the VQA-RAD and PathVQA data sets. Similarly, in image captioning tasks, our model beat the SOTA, suggesting the importance of pretraining with medical data sets. Overall, in 20 different data sets and task combinations, our BioMedBLIP excelled in 15 (75%) out of 20 tasks. BioMedBLIP represents a new SOTA in 15 (75%) out of 20 tasks, and our responses were rated higher in all 20 tasks (P
Subjects
Author keywords
Authors and institutions
- School of Computing, Macquarie University, Sydney, Australia.
- Department of Computer Science, Virginia Tech, Blacksburg, VA, United States.
- Department of Circulation and Medical Imaging, Norwegian University of Science and Technology, Trondheim, Norway.
- Harvard Medical School, Harvard University, Boston, MA, United States.
- Department of Radiology, Boston Children's Hospital, Boston, MA, United States.
ORCID profiles: Usman Naseem, Surendrabikram Thapa, Anum Masood
Cite
APA
Naseem, U., Thapa, S., & Masood, A. (2024). Advancing Accuracy in Multimodal Medical Tasks Through Bootstrapped Language-Image Pretraining (BioMedBLIP): Performance Evaluation Study. JMIR Medical Informatics, 12, e56627. https://doi.org/10.2196/56627
Vancouver
Naseem U, Thapa S, Masood A. Advancing Accuracy in Multimodal Medical Tasks Through Bootstrapped Language-Image Pretraining (BioMedBLIP): Performance Evaluation Study. JMIR Medical Informatics. 2024;12:e56627. doi:10.2196/56627
BibTeX
@article{Naseem2024,
author = {Naseem, Usman and Thapa, Surendrabikram and Masood, Anum},
title = {Advancing Accuracy in Multimodal Medical Tasks Through Bootstrapped Language-Image Pretraining (BioMedBLIP): Performance Evaluation Study},
journal = {JMIR Medical Informatics},
year = {2024},
volume = {12},
pages = {e56627},
doi = {10.2196/56627}
}
EUVIMED