CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers
IEEE Transactions on Intelligent Transportation Systems · 2023-12 · 24 (12) · 14679-14694
DOI: 10.1109/tits.2023.3300537 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Intelligent Transportation Systems, 2023-12. Verfasst von Jiaming Zhang und 5 weiteren Autor:innen. Die Arbeit wurde bisher 697-mal zitiert (Crossref) und nennt 104 Literaturangaben. Gefördert von: Federal Ministry of Labor and Social Affairs (BMAS) through the AccessibleMaps Project, “KIT Future Fields” Project, Ministry of Science, Research and the Arts of Baden-Wurttemberg (MWK) through the Cooperative Graduate School Accessibility through AIBased Assistive Technology.
Abstract (Originalsprache)
Scene understanding based on image segmentation is a crucial component of autonomous vehicles. Pixel-wise semantic segmentation of RGB images can be advanced by exploiting complementary features from the supplementary modality ( ${X}$ -modality). However, covering a wide variety of sensors with a modality-agnostic model remains an unresolved problem due to variations in sensor characteristics among different modalities. Unlike previous modality-specific methods, in this work, we propose a unified fusion framework, CMX, for RGB-X semantic segmentation. To generalize well across different modalities, that often include supplements as well as uncertainties, a unified cross-modal interaction is crucial for modality fusion. Specifically, we design a Cross-Modal Feature Rectification Module (CM-FRM) to calibrate bi-modal features by leveraging the features from one modality to rectify the features of the other modality. With rectified feature pairs, we deploy a Feature Fusion Module (FFM) to perform sufficient exchange of long-range contexts before mixing. To verify CMX, for the first time, we unify five modalities complementary to RGB, i.e., depth, thermal, polarization, event, and LiDAR. Extensive experiments show that CMX generalizes well to diverse multi-modal fusion, achieving state-of-the-art performances on five RGB-Depth benchmarks, as well as RGB-Thermal, RGB-Polarization, and RGB-LiDAR datasets. Besides, to investigate the generalizability to dense-sparse data fusion, we establish an RGB-Event semantic segmentation benchmark based on the EventScape dataset, on which CMX sets the new state-of-the-art. The source code of CMX is publicly available at https://github.com/huaaaliu/RGBX_Semantic_ Segmentation.
Förderung
- Federal Ministry of Labor and Social Affairs (BMAS) through the AccessibleMaps Project
- “KIT Future Fields” Project
- Ministry of Science, Research and the Arts of Baden-Wurttemberg (MWK) through the Cooperative Graduate School Accessibility through AIBased Assistive Technology
- Federal Ministry of Education and Research (BMBF) through the Fellowship within the IFI Program of the German Academic Exchange Service
- HoreKA@KIT Supercomputer Partition
- Hangzhou SurImage Technology Company Ltd
Zitieren
APA
Zhang, J., Liu, H., Yang, K., Hu, X., Liu, R., & Stiefelhagen, R. (2023). CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers. IEEE Transactions on Intelligent Transportation Systems, 24(12), 14679-14694. https://doi.org/10.1109/tits.2023.3300537
Vancouver
Zhang J, Liu H, Yang K, Hu X, Liu R, Stiefelhagen R. CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers. IEEE Transactions on Intelligent Transportation Systems. 2023;24(12):14679-14694. doi:10.1109/tits.2023.3300537
BibTeX
@article{Zhang2023,
author = {Zhang, Jiaming and Liu, Huayao and Yang, Kailun and Hu, Xinxin and Liu, Ruiping and Stiefelhagen, Rainer},
title = {CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers},
journal = {IEEE Transactions on Intelligent Transportation Systems},
year = {2023},
volume = {24},
number = {12},
pages = {14679--14694},
doi = {10.1109/tits.2023.3300537}
}
EUVIMED