Cross-Modal Knowledge Diffusion-Based Generation for Difference-Aware Medical VQA
IEEE Transactions on Image Processing · 2025 · 34 · 2421-2434
DOI: 10.1109/tip.2025.3558446 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Image Processing, 2025. Verfasst von Qika Lin und 5 weiteren Autor:innen. Die Arbeit wurde bisher 16-mal zitiert (Crossref) und nennt 60 Literaturangaben. Gefördert von: National Research Foundation Singapore, Industry Alignment Fund (Cisco-National University of Singapore, Artificial Intelligence (AI) Singapore Program.
Abstract (Originalsprache)
Multimodal medical applications have garnered considerable attention due to their potential to offer comprehensive and robust support for medical assistance. Specifically, within this domain, difference-aware medical Visual Question Answering (VQA) has emerged as a topic of increasing interest that enables the recognition of changes in physical conditions over time when compared to previous states and provides customized suggestions accordingly. However, it is challenging because samples usually exhibit characteristics of complexity, diversity, and inherent noise. Besides, there is a need for multimodal knowledge understanding of the medical domain. The difference-aware setting requiring image comparison further intensifies these situations. To this end, we propose a cross-Modal knowlEdge diffusioN-baseD gEneration netwoRk (MENDER), where the diffusion mechanism with multi-step denoising and knowledge injection from global to local level are employed to tackle the aforementioned challenges, respectively. The diffusion process is to gradually generate answers with the sequence input of questions, random noises for the answer masks and virtual vision prompts of images. The strategy of answer nosing and knowledge cascading is specifically tailored for this task and is implemented during forward and reverse diffusion processes. Moreover, the visual and structure knowledge injection are proposed to learn virtual vision prompts to guide the diffusion process, where the former is realized using a pre-trained medical image-text network and the latter is modeled with spatial and semantic graph structures processed by the heterogeneous graph Transformer models. Experiment results demonstrate the effectiveness of MENDER for difference-aware medical VQA. Furthermore, it also exhibits notable performance in the low-resource setting and conventional medical VQA tasks.
Förderung
- National Research Foundation Singapore
- Industry Alignment Fund (Cisco-National University of Singapore
- Artificial Intelligence (AI) Singapore Program
- National Natural Science Foundation of China
- Social Media Processing (SMP)-Zhipu.AI Large Model Cross-Disciplinary Fund
Zitieren
APA
Lin, Q., He, K., Zhu, Y., Xu, F., Cambria, E., & Feng, M. (2025). Cross-Modal Knowledge Diffusion-Based Generation for Difference-Aware Medical VQA. IEEE Transactions on Image Processing, 34, 2421-2434. https://doi.org/10.1109/tip.2025.3558446
Vancouver
Lin Q, He K, Zhu Y, Xu F, Cambria E, Feng M. Cross-Modal Knowledge Diffusion-Based Generation for Difference-Aware Medical VQA. IEEE Transactions on Image Processing. 2025;34:2421-2434. doi:10.1109/tip.2025.3558446
BibTeX
@article{Lin2025,
author = {Lin, Qika and He, Kai and Zhu, Yifan and Xu, Fangzhi and Cambria, Erik and Feng, Mengling},
title = {Cross-Modal Knowledge Diffusion-Based Generation for Difference-Aware Medical VQA},
journal = {IEEE Transactions on Image Processing},
year = {2025},
volume = {34},
pages = {2421--2434},
doi = {10.1109/tip.2025.3558446}
}
EUVIMED