EUVIMEDEUVIMED entdecken
Crossref

Multi-Modal Feature Pyramid Transformer for RGB-Infrared Object Detection

Yaohui Zhu, Xiaoyu Sun, Miao Wang, Hua Huang

IEEE Transactions on Intelligent Transportation Systems · 2023-09 · 24 (9) · 9984-9995

DOI: 10.1109/tits.2023.3266487 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Intelligent Transportation Systems, 2023-09. Verfasst von Yaohui Zhu und 3 weiteren Autor:innen. Die Arbeit wurde bisher 130-mal zitiert (Crossref) und nennt 62 Literaturangaben. Gefördert von: National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

RGB-Infrared multi-modal object detection utilizes diverse and complementary information, showing some advantages in intelligent transportation field. The main challenge of RGB-Infrared object detection is how to fuse the two modalities. The difficulty of fusion is reflected in two aspects: 1) large visual differences between modalities make it difficult to learn effective complementary features, 2) some misaligned RGB-Infrared images increase the difficulty of fusion. To this end, based on feature pyramid commonly used in object detection, we propose Multi-modal Feature Pyramid Transformer (MFPT) to fuse the two modalities. The proposed MFPT learns semantic and modal complementary information to enhance each modal features via intra-modal feature pyramid transformer and inter-modal feature pyramid transformer. The intra-modal feature pyramid transformer enables features to interact across space and scales, improving the semantic representations of features in each modality. The inter-modal feature pyramid transformer conducts feature interaction between modalities, enabling each modality to learn complementary features from other modalities. Meanwhile, the inter-modal feature pyramid transformer can also learn distance independent dependencies between modalities, which are not sensitive to misaligned images. Furthermore, a local attention mechanism is introduced within different windows into MFPT to achieve efficient correlation between regions of different scales or different modalities. Experimental results on two RGB-Infrared detection datasets demonstrate the proposed method is superior to state-of-the-art methods.

Förderung

  • National Natural Science Foundation of China

Zitieren

APA

Zhu, Y., Sun, X., Wang, M., & Huang, H. (2023). Multi-Modal Feature Pyramid Transformer for RGB-Infrared Object Detection. IEEE Transactions on Intelligent Transportation Systems, 24(9), 9984-9995. https://doi.org/10.1109/tits.2023.3266487

Vancouver

Zhu Y, Sun X, Wang M, Huang H. Multi-Modal Feature Pyramid Transformer for RGB-Infrared Object Detection. IEEE Transactions on Intelligent Transportation Systems. 2023;24(9):9984-9995. doi:10.1109/tits.2023.3266487

BibTeX
@article{Zhu2023,
  author = {Zhu, Yaohui and Sun, Xiaoyu and Wang, Miao and Huang, Hua},
  title = {Multi-Modal Feature Pyramid Transformer for RGB-Infrared Object Detection},
  journal = {IEEE Transactions on Intelligent Transportation Systems},
  year = {2023},
  volume = {24},
  number = {9},
  pages = {9984--9995},
  doi = {10.1109/tits.2023.3266487}
}
Kostenlos starten →