A Cross-Modal Attention-Driven Multi-Sensor Fusion Method for Semantic Segmentation of Point Clouds
Sensors · 14.04.2025 · 25 (8) · 2474
DOI: 10.3390/s25082474 ↗Kurz gefasst
Fachartikel in der Zeitschrift Sensors, online erschienen am 14.04.2025. Verfasst von Huisheng Shi und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Remote Sensing Engineering, Henan College of Surveying and Mapping, Zhengzhou 451464, China. Die Arbeit wurde bisher 17-mal zitiert (Crossref) und nennt 55 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Key Research and Development Program of China, the National Natural Science Foundation of China, Fundamental Research Funds for the Central Universities, Beijing University of Civil Engineering and Architecture.
Abstract (Originalsprache)
To bridge the modality gap between camera images and LiDAR point clouds in autonomous driving systems—a critical challenge exacerbated by current fusion methods’ inability to effectively integrate cross-modal features—we propose the Cross-Modal Fusion (CMF) framework. This attention-driven architecture enables hierarchical multi-sensor data fusion, achieving state-of-the-art performance in semantic segmentation tasks.The CMF framework first projects point clouds onto the camera coordinates through the use of perspective projection to provide spatio-depth information for RGB images. Then, a two-stream feature extraction network is proposed to extract features from the two modalities separately, and multilevel fusion of the two modalities is realized by a residual fusion module (RCF) with cross-modal attention. Finally, we design a perceptual alignment loss that integrates cross-entropy with feature matching terms, effectively minimizing the semantic discrepancy between camera and LiDAR representations during fusion. The experimental results based on the SemanticKITTI and nuScenes benchmark datasets demonstrate that the CMF method achieves mean intersection over union (mIoU) scores of 64.2% and 79.3%, respectively, outperforming existing state-of-the-art methods in regard to accuracy and exhibiting enhanced robustness in regard to complex scenarios. The results of the ablation studies further validate that enhancing the feature interaction and fusion capabilities in semantic segmentation models through cross-modal attention and perceptually guided cross-entropy loss (Pgce) is effective in regard to improving segmentation accuracy and robustness.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Remote Sensing Engineering, Henan College of Surveying and Mapping, Zhengzhou 451464, China.
- Beiqi Foton Motor Co., Ltd., Beijing 102206, China.
- School of Geomatics and Urban Spatial Informatics, Beijing University of Civil Engineering and Architecture, Beijing 102627, China.
- State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430000, China.
ORCID-Profile: Jianghong Zhao
Förderung
- National Key Research and Development Program of China
- the National Natural Science Foundation of China
- Fundamental Research Funds for the Central Universities, Beijing University of Civil Engineering and Architecture
- National Natural Science Foundation of China
- Open Fund Project of State Key Laboratory of Geoinformation Engineering
- Fundamental Research Funds for the Central Universities, Beijing Uni-versity of Civil Engineering and Architecture
- Beijing Natural Science Foundation Project
- Open Research Fund of the State Key Laboratory of Surveying, Mapping and Remote Sensing Information Engineering, Wuhan University
Zitieren
APA
Shi, H., Wang, X., Zhao, J., & Hua, X. (2025). A Cross-Modal Attention-Driven Multi-Sensor Fusion Method for Semantic Segmentation of Point Clouds. Sensors, 25(8), 2474. https://doi.org/10.3390/s25082474
Vancouver
Shi H, Wang X, Zhao J, Hua X. A Cross-Modal Attention-Driven Multi-Sensor Fusion Method for Semantic Segmentation of Point Clouds. Sensors. 2025;25(8):2474. doi:10.3390/s25082474
BibTeX
@article{Shi2025,
author = {Shi, Huisheng and Wang, Xin and Zhao, Jianghong and Hua, Xinnan},
title = {A Cross-Modal Attention-Driven Multi-Sensor Fusion Method for Semantic Segmentation of Point Clouds},
journal = {Sensors},
year = {2025},
volume = {25},
number = {8},
pages = {2474},
doi = {10.3390/s25082474}
}
EUVIMED