Uni-to-Multi Modal Knowledge Distillation for Bidirectional LiDAR-Camera Semantic Segmentation
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-12 · 46 (12) · 11059-11072
DOI: 10.1109/tpami.2024.3451658 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-12. Verfasst von Tianfang Sun und 5 weiteren Autor:innen. Die Arbeit wurde bisher 21-mal zitiert (Crossref) und nennt 67 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Chenguang Program of Shanghai Education Development Foundation and Shanghai Municipal Education Commission, Shanghai Sailing Program.
Abstract (Originalsprache)
Combining LiDAR points and images for robust semantic segmentation has shown great potential. However, the heterogeneity between the two modalities (e.g. the density, the field of view) poses challenges in establishing a bijective mapping between each point and pixel. This modality alignment problem introduces new challenges in network design and data processing for cross-modal methods. Specifically, 1) points that are projected outside the image planes; 2) the complexity of maintaining geometric consistency limits the deployment of many data augmentation techniques. To address these challenges, we propose a cross-modal knowledge imputation and transition approach. First, we introduce a bidirectional feature fusion strategy that imputes missing image features and performs cross-modal fusion simultaneously. This allows us to generate reliable predictions even when images are missing. Second, we propose a Uni-to-Multi modal Knowledge Distillation (U2MKD) framework, leveraging the transfer of informative features from a single-modality teacher to a cross-modality student. This overcomes the issues of augmentation misalignment and enables us to train the student effectively. Extensive experiments on the nuScenes, Waymo, and SemanticKITTI datasets demonstrate the effectiveness of our approach. Notably, our method achieves an 8.3 mIoU gain over the LiDAR-only baseline on the nuScenes validation set and achieves state-of-the-art performance on the three datasets.
Förderung
- National Natural Science Foundation of China
- Chenguang Program of Shanghai Education Development Foundation and Shanghai Municipal Education Commission
- Shanghai Sailing Program
- Natural Science Foundation of Shanghai
- Natural Science Foundation of Chongqing
- CCF-Tencent Rhino-Bird Young Faculty Open Research Fund
Zitieren
APA
Sun, T., Zhang, Z., Tan, X., Peng, Y., Qu, Y., & Xie, Y. (2024). Uni-to-Multi Modal Knowledge Distillation for Bidirectional LiDAR-Camera Semantic Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12), 11059-11072. https://doi.org/10.1109/tpami.2024.3451658
Vancouver
Sun T, Zhang Z, Tan X, Peng Y, Qu Y, Xie Y. Uni-to-Multi Modal Knowledge Distillation for Bidirectional LiDAR-Camera Semantic Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(12):11059-11072. doi:10.1109/tpami.2024.3451658
BibTeX
@article{Sun2024,
author = {Sun, Tianfang and Zhang, Zhizhong and Tan, Xin and Peng, Yong and Qu, Yanyun and Xie, Yuan},
title = {Uni-to-Multi Modal Knowledge Distillation for Bidirectional LiDAR-Camera Semantic Segmentation},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2024},
volume = {46},
number = {12},
pages = {11059--11072},
doi = {10.1109/tpami.2024.3451658}
}
EUVIMED