EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

MDC-RHT: Multi-Modal Medical Image Fusion via Multi-Dimensional Dynamic Convolution and Residual Hybrid Transformer

Wenqing Wang iD, Ji He, Han Liu iD, Wei Yuan iD

Sensors · 21.06.2024 · 24 (13) · 4056

DOI: 10.3390/s24134056 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, online erschienen am 21.06.2024. Verfasst von Wenqing Wang und 3 weiteren Autor:innen. Beteiligte Einrichtung: School of Automation and Information Engineering, Xi'an University of Technology, Xi'an 710048, China. Die Arbeit wurde bisher 27-mal zitiert (Crossref) und nennt 54 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, Natural Science Basic Research Program of Shaanxi.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The fusion of multi-modal medical images has great significance for comprehensive diagnosis and treatment. However, the large differences between the various modalities of medical images make multi-modal medical image fusion a great challenge. This paper proposes a novel multi-scale fusion network based on multi-dimensional dynamic convolution and residual hybrid transformer, which has better capability for feature extraction and context modeling and improves the fusion performance. Specifically, the proposed network exploits multi-dimensional dynamic convolution that introduces four attention mechanisms corresponding to four different dimensions of the convolutional kernel to extract more detailed information. Meanwhile, a residual hybrid transformer is designed, which activates more pixels to participate in the fusion process by channel attention, window attention, and overlapping cross attention, thereby strengthening the long-range dependence between different modes and enhancing the connection of global context information. A loss function, including perceptual loss and structural similarity loss, is designed, where the former enhances the visual reality and perceptual details of the fused image, and the latter enables the model to learn structural textures. The whole network adopts a multi-scale architecture and uses an unsupervised end-to-end method to realize multi-modal image fusion. Finally, our method is tested qualitatively and quantitatively on mainstream datasets. The fusion results indicate that our method achieves high scores in most quantitative indicators and satisfactory performance in visual qualitative analysis.

Schlagwörter

Stichwörter der Autor:innen

deep learningmedical image fusionmulti-dimensional dynamic convolutionresidual hybrid transformer

Autor:innen und Institutionen

  • School of Automation and Information Engineering, Xi'an University of Technology, Xi'an 710048, China.
  • Shaanxi Key Laboratory of Complex System Control and Intelligent Information Processing, Xi'an University of Technology, Xi'an 710048, China.

ORCID-Profile: Wenqing Wang, Han Liu, Wei Yuan

Förderung

  • National Natural Science Foundation of China
  • Natural Science Basic Research Program of Shaanxi

Zitieren

APA

Wang, W., He, J., Liu, H., & Yuan, W. (2024). MDC-RHT: Multi-Modal Medical Image Fusion via Multi-Dimensional Dynamic Convolution and Residual Hybrid Transformer. Sensors, 24(13), 4056. https://doi.org/10.3390/s24134056

Vancouver

Wang W, He J, Liu H, Yuan W. MDC-RHT: Multi-Modal Medical Image Fusion via Multi-Dimensional Dynamic Convolution and Residual Hybrid Transformer. Sensors. 2024;24(13):4056. doi:10.3390/s24134056

BibTeX
@article{Wang2024,
  author = {Wang, Wenqing and He, Ji and Liu, Han and Yuan, Wei},
  title = {MDC-RHT: Multi-Modal Medical Image Fusion via Multi-Dimensional Dynamic Convolution and Residual Hybrid Transformer},
  journal = {Sensors},
  year = {2024},
  volume = {24},
  number = {13},
  pages = {4056},
  doi = {10.3390/s24134056}
}
Kostenlos starten →