EUVIMEDEUVIMED entdecken
Crossref

SAMFlow: Eliminating Any Fragmentation in Optical Flow with Segment Anything Model

Shili Zhou, Ruian He, Weimin Tan, Bo Yan

Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (7) · 7695-7703

DOI: 10.1609/aaai.v38i7.28603 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Shili Zhou und 3 weiteren Autor:innen. Die Arbeit wurde bisher 19-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Optical Flow Estimation aims to find the 2D dense motion field between two frames. Due to the limitation of model structures and training datasets, existing methods often rely too much on local clues and ignore the integrity of objects, resulting in fragmented motion estimation. Through theoretical analysis, we find the pre-trained large vision models are helpful in optical flow estimation, and we notice that the recently famous Segment Anything Model (SAM) demonstrates a strong ability to segment complete objects, which is suitable for solving the fragmentation problem. We thus propose a solution to embed the frozen SAM image encoder into FlowFormer to enhance object perception. To address the challenge of in-depth utilizing SAM in non-segmentation tasks like optical flow estimation, we propose an Optical Flow Task-Specific Adaption scheme, including a Context Fusion Module to fuse the SAM encoder with the optical flow context encoder, and a Context Adaption Module to adapt the SAM features for optical flow task with Learned Task-Specific Embedding. Our proposed SAMFlow model reaches 0.86/2.10 clean/final EPE and 3.55/12.32 EPE/F1-all on Sintel and KITTI-15 training set, surpassing Flowformer by 8.5%/9.9% and 13.2%/16.3%. Furthermore, our model achieves state-of-the-art performance on the Sintel and KITTI-15 benchmarks, ranking #1 among all two-frame methods on Sintel clean pass.

Zitieren

APA

Zhou, S., He, R., Tan, W., & Yan, B. (2024). SAMFlow: Eliminating Any Fragmentation in Optical Flow with Segment Anything Model. Proceedings of the AAAI Conference on Artificial Intelligence, 38(7), 7695-7703. https://doi.org/10.1609/aaai.v38i7.28603

Vancouver

Zhou S, He R, Tan W, Yan B. SAMFlow: Eliminating Any Fragmentation in Optical Flow with Segment Anything Model. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(7):7695-7703. doi:10.1609/aaai.v38i7.28603

BibTeX
@article{Zhou2024,
  author = {Zhou, Shili and He, Ruian and Tan, Weimin and Yan, Bo},
  title = {SAMFlow: Eliminating Any Fragmentation in Optical Flow with Segment Anything Model},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2024},
  volume = {38},
  number = {7},
  pages = {7695--7703},
  doi = {10.1609/aaai.v38i7.28603}
}
Kostenlos starten →