EUVIMEDEUVIMED entdecken
Open Access

Advancing Accuracy in Multimodal Medical Tasks Through Bootstrapped Language-Image Pretraining (BioMedBLIP): Performance Evaluation Study

Usman Naseem iD, Surendrabikram Thapa iD, Anum Masood iD

JMIR Medical Informatics · 05.08.2024 · 12 · e56627

DOI: 10.2196/56627 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Medical Informatics, online erschienen am 05.08.2024. Verfasst von Usman Naseem und 2 weiteren Autor:innen. Beteiligte Einrichtung: School of Computing, Macquarie University, Sydney, Australia. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 35 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Medical image analysis, particularly in the context of visual question answering (VQA) and image captioning, is crucial for accurate diagnosis and educational purposes.

Objective

Our study aims to introduce BioMedBLIP models, fine-tuned for VQA tasks using specialized medical data sets such as Radiology Objects in Context and Medical Information Mart for Intensive Care-Chest X-ray, and evaluate their performance in comparison to the state of the art (SOTA) original Bootstrapping Language-Image Pretraining (BLIP) model.

Methods

We present 9 versions of BioMedBLIP across 3 downstream tasks in various data sets. The models are trained on a varying number of epochs. The findings indicate the strong overall performance of our models. We proposed BioMedBLIP for the VQA generation model, VQA classification model, and BioMedBLIP image caption model. We conducted pretraining in BLIP using medical data sets, producing an adapted BLIP model tailored for medical applications.

Results

In VQA generation tasks, BioMedBLIP models outperformed the SOTA on the Semantically-Labeled Knowledge-Enhanced (SLAKE) data set, VQA in Radiology (VQA-RAD), and Image Cross-Language Evaluation Forum data sets. In VQA classification, our models consistently surpassed the SOTA on the SLAKE data set. Our models also showed competitive performance on the VQA-RAD and PathVQA data sets. Similarly, in image captioning tasks, our model beat the SOTA, suggesting the importance of pretraining with medical data sets. Overall, in 20 different data sets and task combinations, our BioMedBLIP excelled in 15 (75%) out of 20 tasks. BioMedBLIP represents a new SOTA in 15 (75%) out of 20 tasks, and our responses were rated higher in all 20 tasks (P

Schlagwörter

Stichwörter der Autor:innen

BioNLPbiomedical text miningmedical image analysismultimodal modelsvision-language pretraining

Autor:innen und Institutionen

  • School of Computing, Macquarie University, Sydney, Australia.
  • Department of Computer Science, Virginia Tech, Blacksburg, VA, United States.
  • Department of Circulation and Medical Imaging, Norwegian University of Science and Technology, Trondheim, Norway.
  • Harvard Medical School, Harvard University, Boston, MA, United States.
  • Department of Radiology, Boston Children's Hospital, Boston, MA, United States.

ORCID-Profile: Usman Naseem, Surendrabikram Thapa, Anum Masood

Zitieren

APA

Naseem, U., Thapa, S., & Masood, A. (2024). Advancing Accuracy in Multimodal Medical Tasks Through Bootstrapped Language-Image Pretraining (BioMedBLIP): Performance Evaluation Study. JMIR Medical Informatics, 12, e56627. https://doi.org/10.2196/56627

Vancouver

Naseem U, Thapa S, Masood A. Advancing Accuracy in Multimodal Medical Tasks Through Bootstrapped Language-Image Pretraining (BioMedBLIP): Performance Evaluation Study. JMIR Medical Informatics. 2024;12:e56627. doi:10.2196/56627

BibTeX
@article{Naseem2024,
  author = {Naseem, Usman and Thapa, Surendrabikram and Masood, Anum},
  title = {Advancing Accuracy in Multimodal Medical Tasks Through Bootstrapped Language-Image Pretraining (BioMedBLIP): Performance Evaluation Study},
  journal = {JMIR Medical Informatics},
  year = {2024},
  volume = {12},
  pages = {e56627},
  doi = {10.2196/56627}
}
Kostenlos starten →