Multi-View Vision Fusion Network: Can 2D Pre-Trained Model Boost 3D Point Cloud Data-Scarce Learning?
IEEE Transactions on Circuits and Systems for Video Technology · 2024-07 · 34 (7) · 5951-5962
DOI: 10.1109/tcsvt.2023.3343495 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems for Video Technology, 2024-07. Verfasst von Haoyang Peng und 5 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 64 Literaturangaben. Gefördert von: National Natural Science Foundation of China, National Key Research and Development Program of China, Shanghai Natural Science Foundation.
Abstract (Originalsprache)
Point cloud based 3D deep model has wide applications in many applications such as autonomous driving, house robot, etc. Inspired by the recent prompt learning in natural language processing, this work proposes a novel Multi-view Vision Fusion Network (MvNet) for few-shot 3D point cloud classification. MvNet investigates the possibility of leveraging the off-the-shelf 2D pre-trained models to achieve the few-shot classification, which can alleviate the over-dependence issue of the existing baseline models towards the large-scale annotated 3D point cloud data. Specifically, MvNet first encodes a 3D point cloud into multi-view image features for a number of different views. Then, a novel multi-view prompt fusion module is developed to fuse information from different views effectively to bridge the gap between 3D point cloud data and 2D pre-trained models. A set of 2D image prompts can then be derived to better describe the suitable prior knowledge for a large-scale pre-trained image model for few-shot 3D point cloud classification. Extensive experiments on ModelNet, ScanObjectNN, and ShapeNet datasets demonstrate that MvNet achieves new state-of-the-art performance for 3D few-shot point cloud image classification. The source code of this work is available at https://github.com/invictus717/MetaTransformer.
Förderung
- National Natural Science Foundation of China
- National Key Research and Development Program of China
- Shanghai Natural Science Foundation
- Zhejiang Laboratory
- Agency for Science, Technology, and Research (A*STAR) Advanced Manufacturing and Engineering (AME) Programmatic Funding
- RobotHTPO Seed Fund
- Economic Development Board (EDB) Space Technology Development Grant
Zitieren
APA
Peng, H., Li, B., Zhang, B., Chen, X., Chen, T., & Zhu, H. (2024). Multi-View Vision Fusion Network: Can 2D Pre-Trained Model Boost 3D Point Cloud Data-Scarce Learning?. IEEE Transactions on Circuits and Systems for Video Technology, 34(7), 5951-5962. https://doi.org/10.1109/tcsvt.2023.3343495
Vancouver
Peng H, Li B, Zhang B, Chen X, Chen T, Zhu H. Multi-View Vision Fusion Network: Can 2D Pre-Trained Model Boost 3D Point Cloud Data-Scarce Learning?. IEEE Transactions on Circuits and Systems for Video Technology. 2024;34(7):5951-5962. doi:10.1109/tcsvt.2023.3343495
BibTeX
@article{Peng2024,
author = {Peng, Haoyang and Li, Baopu and Zhang, Bo and Chen, Xin and Chen, Tao and Zhu, Hongyuan},
title = {Multi-View Vision Fusion Network: Can 2D Pre-Trained Model Boost 3D Point Cloud Data-Scarce Learning?},
journal = {IEEE Transactions on Circuits and Systems for Video Technology},
year = {2024},
volume = {34},
number = {7},
pages = {5951--5962},
doi = {10.1109/tcsvt.2023.3343495}
}
EUVIMED