EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

MFTransNet: A Multi-Modal Fusion with CNN-Transformer Network for Semantic Segmentation of HSR Remote Sensing Images

Shumeng He, Houqun Yang, Xiaoying Zhang, Xuanyu Li

Mathematics · 01.02.2023 · 11 (3) · 722

DOI: 10.3390/math11030722 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Mathematics, online erschienen am 01.02.2023. Verfasst von Shumeng He und 3 weiteren Autor:innen. Die Arbeit wurde bisher 58-mal zitiert (Crossref) und nennt 48 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Hainan Province Science and Technology Special Fund, Hainan Provincial Natural Science Foundation of China, Haikou Science and Technology Plan Project.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Due to the inherent inter-class similarity and class imbalance of remote sensing images, it is difficult to obtain effective results in single-source semantic segmentation. We consider applying multi-modal data to the task of the semantic segmentation of HSR (high spatial resolution) remote sensing images, and obtain richer semantic information by data fusion to improve the accuracy and efficiency of segmentation. However, it is still a great challenge to discover how to achieve efficient and useful information complementarity based on multi-modal remote sensing image semantic segmentation, so we have to seriously examine the numerous models. Transformer has made remarkable progress in decreasing model complexity and improving scalability and training efficiency in computer vision tasks. Therefore, we introduce Transformer into multi-modal semantic segmentation. In order to cope with the issue that the Transformer model requires a large amount of computing resources, we propose a model, MFTransNet, which combines a CNN (convolutional neural network) and Transformer to realize a lightweight multi-modal semantic segmentation structure. To do this, a small convolutional network is first used for performing preliminary feature extraction. Subsequently, these features are sent to the multi-head feature fusion module to achieve adaptive feature fusion. Finally, the features of different scales are integrated together through a multi-scale decoder. The experimental results demonstrate that MFTransNet achieves the best balance among segmentation accuracy, memory-usage efficiency and inference speed.

Förderung

  • Hainan Province Science and Technology Special Fund
  • Hainan Provincial Natural Science Foundation of China
  • Haikou Science and Technology Plan Project

Zitieren

APA

He, S., Yang, H., Zhang, X., & Li, X. (2023). MFTransNet: A Multi-Modal Fusion with CNN-Transformer Network for Semantic Segmentation of HSR Remote Sensing Images. Mathematics, 11(3), 722. https://doi.org/10.3390/math11030722

Vancouver

He S, Yang H, Zhang X, Li X. MFTransNet: A Multi-Modal Fusion with CNN-Transformer Network for Semantic Segmentation of HSR Remote Sensing Images. Mathematics. 2023;11(3):722. doi:10.3390/math11030722

BibTeX
@article{He2023,
  author = {He, Shumeng and Yang, Houqun and Zhang, Xiaoying and Li, Xuanyu},
  title = {MFTransNet: A Multi-Modal Fusion with CNN-Transformer Network for Semantic Segmentation of HSR Remote Sensing Images},
  journal = {Mathematics},
  year = {2023},
  volume = {11},
  number = {3},
  pages = {722},
  doi = {10.3390/math11030722}
}
Kostenlos starten →