EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Vision–Language Model for Visual Question Answering in Medical Imagery

Yakoub Bazi iD, Mohamad Mahmoud Al Rahhal iD, Laila Bashmal, Mansour Zuair iD

Bioengineering · 20.03.2023 · 10 (3) · 380

DOI: 10.3390/bioengineering10030380 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Bioengineering, online erschienen am 20.03.2023. Verfasst von Yakoub Bazi und 3 weiteren Autor:innen. Beteiligte Einrichtung: Computer Engineering Department, College of Computer and Information Sciences, King Saud University, Riyadh 11543, Saudi Arabia. Die Arbeit wurde bisher 97-mal zitiert (Crossref) und nennt 65 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Deputyship for Research & Innovation, "Ministry of Education" in Saudi Arabia, Deputyship for Research & Innovation, “Ministry of Education”.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In the clinical and healthcare domains, medical images play a critical role. A mature medical visual question answering system (VQA) can improve diagnosis by answering clinical questions presented with a medical image. Despite its enormous potential in the healthcare industry and services, this technology is still in its infancy and is far from practical use. This paper introduces an approach based on a transformer encoder–decoder architecture. Specifically, we extract image features using the vision transformer (ViT) model, and we embed the question using a textual encoder transformer. Then, we concatenate the resulting visual and textual representations and feed them into a multi-modal decoder for generating the answer in an autoregressive way. In the experiments, we validate the proposed model on two VQA datasets for radiology images termed VQA-RAD and PathVQA. The model shows promising results compared to existing solutions. It yields closed and open accuracies of 84.99% and 72.97%, respectively, for VQA-RAD, and 83.86% and 62.37%, respectively, for PathVQA. Other metrics such as the BLUE score showing the alignment between the predicted and true answer sentences are also reported.

Schlagwörter

Stichwörter der Autor:innen

medical visual question answeringtransformervision–language encoders

Autor:innen und Institutionen

  • Computer Engineering Department, College of Computer and Information Sciences, King Saud University, Riyadh 11543, Saudi Arabia.
  • Applied Computer Science Department, College of Applied Computer Science, King Saud University, Riyadh 11543, Saudi Arabia.

ORCID-Profile: Yakoub Bazi, Mohamad Mahmoud Al Rahhal, Mansour Zuair

Förderung

  • Deputyship for Research & Innovation, "Ministry of Education" in Saudi Arabia
  • Deputyship for Research & Innovation, “Ministry of Education”

Zitieren

APA

Bazi, Y., Rahhal, M. M. A., Bashmal, L., & Zuair, M. (2023). Vision–Language Model for Visual Question Answering in Medical Imagery. Bioengineering, 10(3), 380. https://doi.org/10.3390/bioengineering10030380

Vancouver

Bazi Y, Rahhal MMA, Bashmal L, Zuair M. Vision–Language Model for Visual Question Answering in Medical Imagery. Bioengineering. 2023;10(3):380. doi:10.3390/bioengineering10030380

BibTeX
@article{Bazi2023,
  author = {Bazi, Yakoub and Rahhal, Mohamad Mahmoud Al and Bashmal, Laila and Zuair, Mansour},
  title = {Vision–Language Model for Visual Question Answering in Medical Imagery},
  journal = {Bioengineering},
  year = {2023},
  volume = {10},
  number = {3},
  pages = {380},
  doi = {10.3390/bioengineering10030380}
}
Kostenlos starten →