EUVIMEDEUVIMED entdecken
Crossref

Toward Robust LiDAR-Camera Fusion in BEV Space via Mutual Deformable Attention and Temporal Aggregation

Jian Wang, Fan Li, Yi An, Xuchong Zhang, Hongbin Sun

IEEE Transactions on Circuits and Systems for Video Technology · 2024-07 · 34 (7) · 5753-5764

DOI: 10.1109/tcsvt.2024.3366664 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems for Video Technology, 2024-07. Verfasst von Jian Wang und 4 weiteren Autor:innen. Die Arbeit wurde bisher 105-mal zitiert (Crossref) und nennt 60 Literaturangaben. Gefördert von: National Key Research and Development Program of China, Natural Science Basic Research Plan in Shaanxi Province of China, Key Research and Development Project in Shaanxi Province.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

LiDAR and camera are two critical sensors that can provide complementary information for accurate 3D object detection. Most works are devoted to improving the detection performance of fusion models on the clean and well-collected datasets. However, the collected point clouds and images in real scenarios may be corrupted to various degrees due to potential sensor malfunctions, which greatly affects the robustness of the fusion model and poses a threat to safe deployment. In this paper, we first analyze the shortcomings of most fusion detectors, which rely mainly on the LiDAR branch, and the potential of the bird’s eye-view (BEV) paradigm in dealing with partial sensor failures. Based on that, we present a robust LiDAR-camera fusion pipeline in unified BEV space with two novel designs under four typical LiDAR-camera malfunction cases. Specifically, a mutual deformable attention is proposed to dynamically model the spatial feature relationship and reduce the interference caused by the corrupted modality, and a temporal aggregation module is devised to fully utilize the rich information in the temporal domain. Together with the decoupled feature extraction for each modality and holistic BEV space fusion, the proposed detector, termed RobBEV, can work stably regardless of single-modality data corruption. Extensive experiments on the large-scale nuScenes dataset under robust settings demonstrate the effectiveness of our approach.

Förderung

  • National Key Research and Development Program of China
  • Natural Science Basic Research Plan in Shaanxi Province of China
  • Key Research and Development Project in Shaanxi Province

Zitieren

APA

Wang, J., Li, F., An, Y., Zhang, X., & Sun, H. (2024). Toward Robust LiDAR-Camera Fusion in BEV Space via Mutual Deformable Attention and Temporal Aggregation. IEEE Transactions on Circuits and Systems for Video Technology, 34(7), 5753-5764. https://doi.org/10.1109/tcsvt.2024.3366664

Vancouver

Wang J, Li F, An Y, Zhang X, Sun H. Toward Robust LiDAR-Camera Fusion in BEV Space via Mutual Deformable Attention and Temporal Aggregation. IEEE Transactions on Circuits and Systems for Video Technology. 2024;34(7):5753-5764. doi:10.1109/tcsvt.2024.3366664

BibTeX
@article{Wang2024,
  author = {Wang, Jian and Li, Fan and An, Yi and Zhang, Xuchong and Sun, Hongbin},
  title = {Toward Robust LiDAR-Camera Fusion in BEV Space via Mutual Deformable Attention and Temporal Aggregation},
  journal = {IEEE Transactions on Circuits and Systems for Video Technology},
  year = {2024},
  volume = {34},
  number = {7},
  pages = {5753--5764},
  doi = {10.1109/tcsvt.2024.3366664}
}
Kostenlos starten →