Advancing Accuracy in Multimodal Medical Tasks Through Bootstrapped Language-Image Pretraining (BioMedBLIP): Performance Evaluation Study
JMIR Medical Informatics · 05.08.2024 · 12 · e56627
DOI: 10.2196/56627 ↗Kurz gefasst
Fachartikel in der Zeitschrift JMIR Medical Informatics, online erschienen am 05.08.2024. Verfasst von Usman Naseem und 2 weiteren Autor:innen. Beteiligte Einrichtung: School of Computing, Macquarie University, Sydney, Australia. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 35 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Medical image analysis, particularly in the context of visual question answering (VQA) and image captioning, is crucial for accurate diagnosis and educational purposes.
Objective
Our study aims to introduce BioMedBLIP models, fine-tuned for VQA tasks using specialized medical data sets such as Radiology Objects in Context and Medical Information Mart for Intensive Care-Chest X-ray, and evaluate their performance in comparison to the state of the art (SOTA) original Bootstrapping Language-Image Pretraining (BLIP) model.
Methods
We present 9 versions of BioMedBLIP across 3 downstream tasks in various data sets. The models are trained on a varying number of epochs. The findings indicate the strong overall performance of our models. We proposed BioMedBLIP for the VQA generation model, VQA classification model, and BioMedBLIP image caption model. We conducted pretraining in BLIP using medical data sets, producing an adapted BLIP model tailored for medical applications.
Results
In VQA generation tasks, BioMedBLIP models outperformed the SOTA on the Semantically-Labeled Knowledge-Enhanced (SLAKE) data set, VQA in Radiology (VQA-RAD), and Image Cross-Language Evaluation Forum data sets. In VQA classification, our models consistently surpassed the SOTA on the SLAKE data set. Our models also showed competitive performance on the VQA-RAD and PathVQA data sets. Similarly, in image captioning tasks, our model beat the SOTA, suggesting the importance of pretraining with medical data sets. Overall, in 20 different data sets and task combinations, our BioMedBLIP excelled in 15 (75%) out of 20 tasks. BioMedBLIP represents a new SOTA in 15 (75%) out of 20 tasks, and our responses were rated higher in all 20 tasks (P
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- School of Computing, Macquarie University, Sydney, Australia.
- Department of Computer Science, Virginia Tech, Blacksburg, VA, United States.
- Department of Circulation and Medical Imaging, Norwegian University of Science and Technology, Trondheim, Norway.
- Harvard Medical School, Harvard University, Boston, MA, United States.
- Department of Radiology, Boston Children's Hospital, Boston, MA, United States.
ORCID-Profile: Usman Naseem, Surendrabikram Thapa, Anum Masood
Zitieren
APA
Naseem, U., Thapa, S., & Masood, A. (2024). Advancing Accuracy in Multimodal Medical Tasks Through Bootstrapped Language-Image Pretraining (BioMedBLIP): Performance Evaluation Study. JMIR Medical Informatics, 12, e56627. https://doi.org/10.2196/56627
Vancouver
Naseem U, Thapa S, Masood A. Advancing Accuracy in Multimodal Medical Tasks Through Bootstrapped Language-Image Pretraining (BioMedBLIP): Performance Evaluation Study. JMIR Medical Informatics. 2024;12:e56627. doi:10.2196/56627
BibTeX
@article{Naseem2024,
author = {Naseem, Usman and Thapa, Surendrabikram and Masood, Anum},
title = {Advancing Accuracy in Multimodal Medical Tasks Through Bootstrapped Language-Image Pretraining (BioMedBLIP): Performance Evaluation Study},
journal = {JMIR Medical Informatics},
year = {2024},
volume = {12},
pages = {e56627},
doi = {10.2196/56627}
}
EUVIMED