Toward Robust LiDAR-Camera Fusion in BEV Space via Mutual Deformable Attention and Temporal Aggregation
IEEE Transactions on Circuits and Systems for Video Technology · 2024-07 · 34 (7) · 5753-5764
DOI: 10.1109/tcsvt.2024.3366664 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems for Video Technology, 2024-07. Verfasst von Jian Wang und 4 weiteren Autor:innen. Die Arbeit wurde bisher 105-mal zitiert (Crossref) und nennt 60 Literaturangaben. Gefördert von: National Key Research and Development Program of China, Natural Science Basic Research Plan in Shaanxi Province of China, Key Research and Development Project in Shaanxi Province.
Abstract (Originalsprache)
LiDAR and camera are two critical sensors that can provide complementary information for accurate 3D object detection. Most works are devoted to improving the detection performance of fusion models on the clean and well-collected datasets. However, the collected point clouds and images in real scenarios may be corrupted to various degrees due to potential sensor malfunctions, which greatly affects the robustness of the fusion model and poses a threat to safe deployment. In this paper, we first analyze the shortcomings of most fusion detectors, which rely mainly on the LiDAR branch, and the potential of the bird’s eye-view (BEV) paradigm in dealing with partial sensor failures. Based on that, we present a robust LiDAR-camera fusion pipeline in unified BEV space with two novel designs under four typical LiDAR-camera malfunction cases. Specifically, a mutual deformable attention is proposed to dynamically model the spatial feature relationship and reduce the interference caused by the corrupted modality, and a temporal aggregation module is devised to fully utilize the rich information in the temporal domain. Together with the decoupled feature extraction for each modality and holistic BEV space fusion, the proposed detector, termed RobBEV, can work stably regardless of single-modality data corruption. Extensive experiments on the large-scale nuScenes dataset under robust settings demonstrate the effectiveness of our approach.
Förderung
- National Key Research and Development Program of China
- Natural Science Basic Research Plan in Shaanxi Province of China
- Key Research and Development Project in Shaanxi Province
Zitieren
APA
Wang, J., Li, F., An, Y., Zhang, X., & Sun, H. (2024). Toward Robust LiDAR-Camera Fusion in BEV Space via Mutual Deformable Attention and Temporal Aggregation. IEEE Transactions on Circuits and Systems for Video Technology, 34(7), 5753-5764. https://doi.org/10.1109/tcsvt.2024.3366664
Vancouver
Wang J, Li F, An Y, Zhang X, Sun H. Toward Robust LiDAR-Camera Fusion in BEV Space via Mutual Deformable Attention and Temporal Aggregation. IEEE Transactions on Circuits and Systems for Video Technology. 2024;34(7):5753-5764. doi:10.1109/tcsvt.2024.3366664
BibTeX
@article{Wang2024,
author = {Wang, Jian and Li, Fan and An, Yi and Zhang, Xuchong and Sun, Hongbin},
title = {Toward Robust LiDAR-Camera Fusion in BEV Space via Mutual Deformable Attention and Temporal Aggregation},
journal = {IEEE Transactions on Circuits and Systems for Video Technology},
year = {2024},
volume = {34},
number = {7},
pages = {5753--5764},
doi = {10.1109/tcsvt.2024.3366664}
}
EUVIMED