Mutually Beneficial Transformer for Multimodal Data Fusion
IEEE Transactions on Circuits and Systems for Video Technology · 2023-12 · 33 (12) · 7466-7479
DOI: 10.1109/tcsvt.2023.3274545 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems for Video Technology, 2023-12. Verfasst von Jinping Wang und 1 weiteren Autor:innen. Die Arbeit wurde bisher 52-mal zitiert (Crossref) und nennt 65 Literaturangaben. Gefördert von: Key-Area Research and Development Program of Guangdong Province, Southern Marine Science and Engineering Guangdong Laboratory.
Abstract (Originalsprache)
Multimodal feature fusion representation, e.g., hyperspectral image and light detection and ranging (HSI-LiDAR) fusion, is an essential topic for fusion perception. However, existing networks tend to employ mandatory feature stacking or local context fusion strategies between multiple modalities, ignoring the power of globally mutual-guided feature transmission. Therefore, this paper develops a mutually beneficial transformer method for multimodal data fusion (MBFormer), which contains the following steps. First, a spatial constraint-based self-attention (SCS) module. In this module, spectralwise attention and a spatialwise convolution are applied to HSI and LiDAR data individually, and then a spatial guide mask generated from LiDAR elevation information is used as an agent to bridge with HSI for spatial feature constraints. Second, a channel diversity-based transformer (CDT) module. On the basis of local spectral embedding explorations, an adaptive token-mixer mechanism is conducted on the groupwise classification token of HSI and individual LiDAR data for global information connectivity and transitivity. At last, the selected features are embedded into a classification layer for the final result calculation. Experimental results show that the proposed MBFormer can obtain 97.76% and 98.62% classification accuracies on Houston and Trento datasets, respectively, indicating the advantages and competitiveness of the MBFormer over the compared state-of-the-art methods.
Förderung
- Key-Area Research and Development Program of Guangdong Province
- Southern Marine Science and Engineering Guangdong Laboratory
Zitieren
APA
Wang, J., & Tan, X. (2023). Mutually Beneficial Transformer for Multimodal Data Fusion. IEEE Transactions on Circuits and Systems for Video Technology, 33(12), 7466-7479. https://doi.org/10.1109/tcsvt.2023.3274545
Vancouver
Wang J, Tan X. Mutually Beneficial Transformer for Multimodal Data Fusion. IEEE Transactions on Circuits and Systems for Video Technology. 2023;33(12):7466-7479. doi:10.1109/tcsvt.2023.3274545
BibTeX
@article{Wang2023,
author = {Wang, Jinping and Tan, Xiaojun},
title = {Mutually Beneficial Transformer for Multimodal Data Fusion},
journal = {IEEE Transactions on Circuits and Systems for Video Technology},
year = {2023},
volume = {33},
number = {12},
pages = {7466--7479},
doi = {10.1109/tcsvt.2023.3274545}
}
EUVIMED