Learning Optical Flow and Scene Flow With Bidirectional Camera-LiDAR Fusion
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-04 · 46 (4) · 2378-2395
DOI: 10.1109/tpami.2023.3330866 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-04. Verfasst von Haisong Liu und 4 weiteren Autor:innen. Die Arbeit wurde bisher 24-mal zitiert (Crossref) und nennt 76 Literaturangaben. Gefördert von: National Key R&D Program of China, National Natural Science Foundation of China, Fundamental Research Funds for the Central Universities.
Abstract (Originalsprache)
In this paper, we study the problem of jointly estimating the optical flow and scene flow from synchronized 2D and 3D data. Previous methods either employ a complex pipeline that splits the joint task into independent stages, or fuse 2D and 3D information in an "early-fusion" or "late-fusion" manner. Such one-size-fits-all approaches suffer from a dilemma of failing to fully utilize the characteristic of each modality or to maximize the inter-modality complementarity. To address the problem, we propose a novel end-to-end framework, which consists of 2D and 3D branches with multiple bidirectional fusion connections between them in specific layers. Different from previous work, we apply a point-based 3D branch to extract the LiDAR features, as it preserves the geometric structure of point clouds. To fuse dense image features and sparse point features, we propose a learnable operator named bidirectional camera-LiDAR fusion module (Bi-CLFM). We instantiate two types of the bidirectional fusion pipeline, one based on the pyramidal coarse-to-fine architecture (dubbed CamLiPWC), and the other one based on the recurrent all-pairs field transforms (dubbed CamLiRAFT). On FlyingThings3D, both CamLiPWC and CamLiRAFT surpass all existing methods and achieve up to a 47.9% reduction in 3D end-point-error from the best published result. Our best-performing model, CamLiRAFT, achieves an error of 4.26% on the KITTI Scene Flow benchmark, ranking 1st among all submissions with much fewer parameters. Besides, our methods have strong generalization performance and the ability to handle non-rigid motion.
Förderung
- National Key R&D Program of China
- National Natural Science Foundation of China
- Fundamental Research Funds for the Central Universities
- Collaborative Innovation Center of Novel Software Technology and Industrialization
Zitieren
APA
Liu, H., Lu, T., Xu, Y., Liu, J., & Wang, L. (2024). Learning Optical Flow and Scene Flow With Bidirectional Camera-LiDAR Fusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(4), 2378-2395. https://doi.org/10.1109/tpami.2023.3330866
Vancouver
Liu H, Lu T, Xu Y, Liu J, Wang L. Learning Optical Flow and Scene Flow With Bidirectional Camera-LiDAR Fusion. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(4):2378-2395. doi:10.1109/tpami.2023.3330866
BibTeX
@article{Liu2024,
author = {Liu, Haisong and Lu, Tao and Xu, Yihui and Liu, Jia and Wang, Limin},
title = {Learning Optical Flow and Scene Flow With Bidirectional Camera-LiDAR Fusion},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2024},
volume = {46},
number = {4},
pages = {2378--2395},
doi = {10.1109/tpami.2023.3330866}
}
EUVIMED