Multi-Modal Feature Pyramid Transformer for RGB-Infrared Object Detection
IEEE Transactions on Intelligent Transportation Systems · 2023-09 · 24 (9) · 9984-9995
DOI: 10.1109/tits.2023.3266487 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Intelligent Transportation Systems, 2023-09. Verfasst von Yaohui Zhu und 3 weiteren Autor:innen. Die Arbeit wurde bisher 130-mal zitiert (Crossref) und nennt 62 Literaturangaben. Gefördert von: National Natural Science Foundation of China.
Abstract (Originalsprache)
RGB-Infrared multi-modal object detection utilizes diverse and complementary information, showing some advantages in intelligent transportation field. The main challenge of RGB-Infrared object detection is how to fuse the two modalities. The difficulty of fusion is reflected in two aspects: 1) large visual differences between modalities make it difficult to learn effective complementary features, 2) some misaligned RGB-Infrared images increase the difficulty of fusion. To this end, based on feature pyramid commonly used in object detection, we propose Multi-modal Feature Pyramid Transformer (MFPT) to fuse the two modalities. The proposed MFPT learns semantic and modal complementary information to enhance each modal features via intra-modal feature pyramid transformer and inter-modal feature pyramid transformer. The intra-modal feature pyramid transformer enables features to interact across space and scales, improving the semantic representations of features in each modality. The inter-modal feature pyramid transformer conducts feature interaction between modalities, enabling each modality to learn complementary features from other modalities. Meanwhile, the inter-modal feature pyramid transformer can also learn distance independent dependencies between modalities, which are not sensitive to misaligned images. Furthermore, a local attention mechanism is introduced within different windows into MFPT to achieve efficient correlation between regions of different scales or different modalities. Experimental results on two RGB-Infrared detection datasets demonstrate the proposed method is superior to state-of-the-art methods.
Förderung
- National Natural Science Foundation of China
Zitieren
APA
Zhu, Y., Sun, X., Wang, M., & Huang, H. (2023). Multi-Modal Feature Pyramid Transformer for RGB-Infrared Object Detection. IEEE Transactions on Intelligent Transportation Systems, 24(9), 9984-9995. https://doi.org/10.1109/tits.2023.3266487
Vancouver
Zhu Y, Sun X, Wang M, Huang H. Multi-Modal Feature Pyramid Transformer for RGB-Infrared Object Detection. IEEE Transactions on Intelligent Transportation Systems. 2023;24(9):9984-9995. doi:10.1109/tits.2023.3266487
BibTeX
@article{Zhu2023,
author = {Zhu, Yaohui and Sun, Xiaoyu and Wang, Miao and Huang, Hua},
title = {Multi-Modal Feature Pyramid Transformer for RGB-Infrared Object Detection},
journal = {IEEE Transactions on Intelligent Transportation Systems},
year = {2023},
volume = {24},
number = {9},
pages = {9984--9995},
doi = {10.1109/tits.2023.3266487}
}
EUVIMED