Spatial-frequency attention-based optical and scene flow with cross-modal knowledge distillation
The Visual Computer · 2025-04 · 41 (6) · 4183-4198
DOI: 10.1007/s00371-024-03654-2 ↗Kurz gefasst
Fachartikel in der Zeitschrift The Visual Computer, online erschienen am 09.10.2024. Verfasst von Youjie Zhou und 4 weiteren Autor:innen. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 44 Literaturangaben. Gefördert von: China government project, Key Technology R&D Program of Shandong.
Abstract (Originalsprache)
This paper studies the problem of multimodal fusion for optical and scene flow from RGB and depth images, or point clouds. Previous methods fuse multimodal information in “early-fusion” or “late-fusion” strategies, in which an attention mechanism is employed to address the problem of optical and scene flow estimation when RGB information is unreliable. Such attentive approaches either suffer from substantial computational and time complexities or lose the inherent characteristics of features due to downsampling. To address this issue, we propose a novel multimodal fusion approach named SFRAFT, which utilizes Fourier transform to build the spatial-frequency domain transformed self-attention and cross-attention. With the novel attentive mechanism, our approach can extract informative features more efficiently and effectively. We further enhance information exchange between the two modalities by incorporating multi-scale knowledge distillation. Experimental results on Flythings3D and KITTI show that our SFRAFT achieves the best performance with low computational and time complexity. We also prove the strong ability of our approach for flow estimation on our real-world dataset. We release the code and datasets at https://doi.org/10.5281/zenodo.12697968.
Förderung
- China government project
- Key Technology R&D Program of Shandong
Zitieren
APA
Zhou, Y., Jiao, R., Tao, Z., Liang, X., & Wan, Y. (2025). Spatial-frequency attention-based optical and scene flow with cross-modal knowledge distillation. The Visual Computer, 41(6), 4183-4198. https://doi.org/10.1007/s00371-024-03654-2
Vancouver
Zhou Y, Jiao R, Tao Z, Liang X, Wan Y. Spatial-frequency attention-based optical and scene flow with cross-modal knowledge distillation. The Visual Computer. 2025;41(6):4183-4198. doi:10.1007/s00371-024-03654-2
BibTeX
@article{Zhou2025,
author = {Zhou, Youjie and Jiao, Runyu and Tao, Zhonghan and Liang, Xichang and Wan, Yi},
title = {Spatial-frequency attention-based optical and scene flow with cross-modal knowledge distillation},
journal = {The Visual Computer},
year = {2025},
volume = {41},
number = {6},
pages = {4183--4198},
doi = {10.1007/s00371-024-03654-2}
}
EUVIMED