EUVIMEDEUVIMED entdecken
Crossref

Cross-Modal Knowledge Diffusion-Based Generation for Difference-Aware Medical VQA

Qika Lin, Kai He, Yifan Zhu, Fangzhi Xu, Erik Cambria, Mengling Feng

IEEE Transactions on Image Processing · 2025 · 34 · 2421-2434

DOI: 10.1109/tip.2025.3558446 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Image Processing, 2025. Verfasst von Qika Lin und 5 weiteren Autor:innen. Die Arbeit wurde bisher 16-mal zitiert (Crossref) und nennt 60 Literaturangaben. Gefördert von: National Research Foundation Singapore, Industry Alignment Fund (Cisco-National University of Singapore, Artificial Intelligence (AI) Singapore Program.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Multimodal medical applications have garnered considerable attention due to their potential to offer comprehensive and robust support for medical assistance. Specifically, within this domain, difference-aware medical Visual Question Answering (VQA) has emerged as a topic of increasing interest that enables the recognition of changes in physical conditions over time when compared to previous states and provides customized suggestions accordingly. However, it is challenging because samples usually exhibit characteristics of complexity, diversity, and inherent noise. Besides, there is a need for multimodal knowledge understanding of the medical domain. The difference-aware setting requiring image comparison further intensifies these situations. To this end, we propose a cross-Modal knowlEdge diffusioN-baseD gEneration netwoRk (MENDER), where the diffusion mechanism with multi-step denoising and knowledge injection from global to local level are employed to tackle the aforementioned challenges, respectively. The diffusion process is to gradually generate answers with the sequence input of questions, random noises for the answer masks and virtual vision prompts of images. The strategy of answer nosing and knowledge cascading is specifically tailored for this task and is implemented during forward and reverse diffusion processes. Moreover, the visual and structure knowledge injection are proposed to learn virtual vision prompts to guide the diffusion process, where the former is realized using a pre-trained medical image-text network and the latter is modeled with spatial and semantic graph structures processed by the heterogeneous graph Transformer models. Experiment results demonstrate the effectiveness of MENDER for difference-aware medical VQA. Furthermore, it also exhibits notable performance in the low-resource setting and conventional medical VQA tasks.

Förderung

  • National Research Foundation Singapore
  • Industry Alignment Fund (Cisco-National University of Singapore
  • Artificial Intelligence (AI) Singapore Program
  • National Natural Science Foundation of China
  • Social Media Processing (SMP)-Zhipu.AI Large Model Cross-Disciplinary Fund

Zitieren

APA

Lin, Q., He, K., Zhu, Y., Xu, F., Cambria, E., & Feng, M. (2025). Cross-Modal Knowledge Diffusion-Based Generation for Difference-Aware Medical VQA. IEEE Transactions on Image Processing, 34, 2421-2434. https://doi.org/10.1109/tip.2025.3558446

Vancouver

Lin Q, He K, Zhu Y, Xu F, Cambria E, Feng M. Cross-Modal Knowledge Diffusion-Based Generation for Difference-Aware Medical VQA. IEEE Transactions on Image Processing. 2025;34:2421-2434. doi:10.1109/tip.2025.3558446

BibTeX
@article{Lin2025,
  author = {Lin, Qika and He, Kai and Zhu, Yifan and Xu, Fangzhi and Cambria, Erik and Feng, Mengling},
  title = {Cross-Modal Knowledge Diffusion-Based Generation for Difference-Aware Medical VQA},
  journal = {IEEE Transactions on Image Processing},
  year = {2025},
  volume = {34},
  pages = {2421--2434},
  doi = {10.1109/tip.2025.3558446}
}
Kostenlos starten →