EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

A Cross-Modal Attention-Driven Multi-Sensor Fusion Method for Semantic Segmentation of Point Clouds

Huisheng Shi, Xin Wang, Jianghong Zhao iD, Xinnan Hua

Sensors · 14.04.2025 · 25 (8) · 2474

DOI: 10.3390/s25082474 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, online erschienen am 14.04.2025. Verfasst von Huisheng Shi und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Remote Sensing Engineering, Henan College of Surveying and Mapping, Zhengzhou 451464, China. Die Arbeit wurde bisher 17-mal zitiert (Crossref) und nennt 55 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Key Research and Development Program of China, the National Natural Science Foundation of China, Fundamental Research Funds for the Central Universities, Beijing University of Civil Engineering and Architecture.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

To bridge the modality gap between camera images and LiDAR point clouds in autonomous driving systems—a critical challenge exacerbated by current fusion methods’ inability to effectively integrate cross-modal features—we propose the Cross-Modal Fusion (CMF) framework. This attention-driven architecture enables hierarchical multi-sensor data fusion, achieving state-of-the-art performance in semantic segmentation tasks.The CMF framework first projects point clouds onto the camera coordinates through the use of perspective projection to provide spatio-depth information for RGB images. Then, a two-stream feature extraction network is proposed to extract features from the two modalities separately, and multilevel fusion of the two modalities is realized by a residual fusion module (RCF) with cross-modal attention. Finally, we design a perceptual alignment loss that integrates cross-entropy with feature matching terms, effectively minimizing the semantic discrepancy between camera and LiDAR representations during fusion. The experimental results based on the SemanticKITTI and nuScenes benchmark datasets demonstrate that the CMF method achieves mean intersection over union (mIoU) scores of 64.2% and 79.3%, respectively, outperforming existing state-of-the-art methods in regard to accuracy and exhibiting enhanced robustness in regard to complex scenarios. The results of the ablation studies further validate that enhancing the feature interaction and fusion capabilities in semantic segmentation models through cross-modal attention and perceptually guided cross-entropy loss (Pgce) is effective in regard to improving segmentation accuracy and robustness.

Schlagwörter

Stichwörter der Autor:innen

cross-modal fusiondeep learningimagemulti-sensorpoint cloudsemantic segmentation

Autor:innen und Institutionen

  • Department of Remote Sensing Engineering, Henan College of Surveying and Mapping, Zhengzhou 451464, China.
  • Beiqi Foton Motor Co., Ltd., Beijing 102206, China.
  • School of Geomatics and Urban Spatial Informatics, Beijing University of Civil Engineering and Architecture, Beijing 102627, China.
  • State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan 430000, China.

ORCID-Profile: Jianghong Zhao

Förderung

  • National Key Research and Development Program of China
  • the National Natural Science Foundation of China
  • Fundamental Research Funds for the Central Universities, Beijing University of Civil Engineering and Architecture
  • National Natural Science Foundation of China
  • Open Fund Project of State Key Laboratory of Geoinformation Engineering
  • Fundamental Research Funds for the Central Universities, Beijing Uni-versity of Civil Engineering and Architecture
  • Beijing Natural Science Foundation Project
  • Open Research Fund of the State Key Laboratory of Surveying, Mapping and Remote Sensing Information Engineering, Wuhan University

Zitieren

APA

Shi, H., Wang, X., Zhao, J., & Hua, X. (2025). A Cross-Modal Attention-Driven Multi-Sensor Fusion Method for Semantic Segmentation of Point Clouds. Sensors, 25(8), 2474. https://doi.org/10.3390/s25082474

Vancouver

Shi H, Wang X, Zhao J, Hua X. A Cross-Modal Attention-Driven Multi-Sensor Fusion Method for Semantic Segmentation of Point Clouds. Sensors. 2025;25(8):2474. doi:10.3390/s25082474

BibTeX
@article{Shi2025,
  author = {Shi, Huisheng and Wang, Xin and Zhao, Jianghong and Hua, Xinnan},
  title = {A Cross-Modal Attention-Driven Multi-Sensor Fusion Method for Semantic Segmentation of Point Clouds},
  journal = {Sensors},
  year = {2025},
  volume = {25},
  number = {8},
  pages = {2474},
  doi = {10.3390/s25082474}
}
Kostenlos starten →