MDC-RHT: Multi-Modal Medical Image Fusion via Multi-Dimensional Dynamic Convolution and Residual Hybrid Transformer
Sensors · 21.06.2024 · 24 (13) · 4056
DOI: 10.3390/s24134056 ↗Kurz gefasst
Fachartikel in der Zeitschrift Sensors, online erschienen am 21.06.2024. Verfasst von Wenqing Wang und 3 weiteren Autor:innen. Beteiligte Einrichtung: School of Automation and Information Engineering, Xi'an University of Technology, Xi'an 710048, China. Die Arbeit wurde bisher 27-mal zitiert (Crossref) und nennt 54 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, Natural Science Basic Research Program of Shaanxi.
Abstract (Originalsprache)
The fusion of multi-modal medical images has great significance for comprehensive diagnosis and treatment. However, the large differences between the various modalities of medical images make multi-modal medical image fusion a great challenge. This paper proposes a novel multi-scale fusion network based on multi-dimensional dynamic convolution and residual hybrid transformer, which has better capability for feature extraction and context modeling and improves the fusion performance. Specifically, the proposed network exploits multi-dimensional dynamic convolution that introduces four attention mechanisms corresponding to four different dimensions of the convolutional kernel to extract more detailed information. Meanwhile, a residual hybrid transformer is designed, which activates more pixels to participate in the fusion process by channel attention, window attention, and overlapping cross attention, thereby strengthening the long-range dependence between different modes and enhancing the connection of global context information. A loss function, including perceptual loss and structural similarity loss, is designed, where the former enhances the visual reality and perceptual details of the fused image, and the latter enables the model to learn structural textures. The whole network adopts a multi-scale architecture and uses an unsupervised end-to-end method to realize multi-modal image fusion. Finally, our method is tested qualitatively and quantitatively on mainstream datasets. The fusion results indicate that our method achieves high scores in most quantitative indicators and satisfactory performance in visual qualitative analysis.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- School of Automation and Information Engineering, Xi'an University of Technology, Xi'an 710048, China.
- Shaanxi Key Laboratory of Complex System Control and Intelligent Information Processing, Xi'an University of Technology, Xi'an 710048, China.
ORCID-Profile: Wenqing Wang, Han Liu, Wei Yuan
Förderung
- National Natural Science Foundation of China
- Natural Science Basic Research Program of Shaanxi
Zitieren
APA
Wang, W., He, J., Liu, H., & Yuan, W. (2024). MDC-RHT: Multi-Modal Medical Image Fusion via Multi-Dimensional Dynamic Convolution and Residual Hybrid Transformer. Sensors, 24(13), 4056. https://doi.org/10.3390/s24134056
Vancouver
Wang W, He J, Liu H, Yuan W. MDC-RHT: Multi-Modal Medical Image Fusion via Multi-Dimensional Dynamic Convolution and Residual Hybrid Transformer. Sensors. 2024;24(13):4056. doi:10.3390/s24134056
BibTeX
@article{Wang2024,
author = {Wang, Wenqing and He, Ji and Liu, Han and Yuan, Wei},
title = {MDC-RHT: Multi-Modal Medical Image Fusion via Multi-Dimensional Dynamic Convolution and Residual Hybrid Transformer},
journal = {Sensors},
year = {2024},
volume = {24},
number = {13},
pages = {4056},
doi = {10.3390/s24134056}
}
EUVIMED