MFTransNet: A Multi-Modal Fusion with CNN-Transformer Network for Semantic Segmentation of HSR Remote Sensing Images
Mathematics · 01.02.2023 · 11 (3) · 722
DOI: 10.3390/math11030722 ↗Kurz gefasst
Fachartikel in der Zeitschrift Mathematics, online erschienen am 01.02.2023. Verfasst von Shumeng He und 3 weiteren Autor:innen. Die Arbeit wurde bisher 58-mal zitiert (Crossref) und nennt 48 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Hainan Province Science and Technology Special Fund, Hainan Provincial Natural Science Foundation of China, Haikou Science and Technology Plan Project.
Abstract (Originalsprache)
Due to the inherent inter-class similarity and class imbalance of remote sensing images, it is difficult to obtain effective results in single-source semantic segmentation. We consider applying multi-modal data to the task of the semantic segmentation of HSR (high spatial resolution) remote sensing images, and obtain richer semantic information by data fusion to improve the accuracy and efficiency of segmentation. However, it is still a great challenge to discover how to achieve efficient and useful information complementarity based on multi-modal remote sensing image semantic segmentation, so we have to seriously examine the numerous models. Transformer has made remarkable progress in decreasing model complexity and improving scalability and training efficiency in computer vision tasks. Therefore, we introduce Transformer into multi-modal semantic segmentation. In order to cope with the issue that the Transformer model requires a large amount of computing resources, we propose a model, MFTransNet, which combines a CNN (convolutional neural network) and Transformer to realize a lightweight multi-modal semantic segmentation structure. To do this, a small convolutional network is first used for performing preliminary feature extraction. Subsequently, these features are sent to the multi-head feature fusion module to achieve adaptive feature fusion. Finally, the features of different scales are integrated together through a multi-scale decoder. The experimental results demonstrate that MFTransNet achieves the best balance among segmentation accuracy, memory-usage efficiency and inference speed.
Förderung
- Hainan Province Science and Technology Special Fund
- Hainan Provincial Natural Science Foundation of China
- Haikou Science and Technology Plan Project
Zitieren
APA
He, S., Yang, H., Zhang, X., & Li, X. (2023). MFTransNet: A Multi-Modal Fusion with CNN-Transformer Network for Semantic Segmentation of HSR Remote Sensing Images. Mathematics, 11(3), 722. https://doi.org/10.3390/math11030722
Vancouver
He S, Yang H, Zhang X, Li X. MFTransNet: A Multi-Modal Fusion with CNN-Transformer Network for Semantic Segmentation of HSR Remote Sensing Images. Mathematics. 2023;11(3):722. doi:10.3390/math11030722
BibTeX
@article{He2023,
author = {He, Shumeng and Yang, Houqun and Zhang, Xiaoying and Li, Xuanyu},
title = {MFTransNet: A Multi-Modal Fusion with CNN-Transformer Network for Semantic Segmentation of HSR Remote Sensing Images},
journal = {Mathematics},
year = {2023},
volume = {11},
number = {3},
pages = {722},
doi = {10.3390/math11030722}
}
EUVIMED