CrossFormer++: A Versatile Vision Transformer Hinging on Cross-Scale Attention
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-05 · 46 (5) · 3123-3136
DOI: 10.1109/tpami.2023.3341806 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-05. Verfasst von Wenxiao Wang und 7 weiteren Autor:innen. Die Arbeit wurde bisher 276-mal zitiert (Crossref) und nennt 58 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Key R&D Program of Zhejiang Province, China, Yongjiang Talent Introduction Programme.
Abstract (Originalsprache)
While features of different scales are perceptually important to visual inputs, existing vision transformers do not yet take advantage of them explicitly. To this end, we first propose a cross-scale vision transformer, CrossFormer. It introduces a cross-scale embedding layer (CEL) and a long-short distance attention (LSDA). On the one hand, CEL blends each token with multiple patches of different scales, providing the self-attention module itself with cross-scale features. On the other hand, LSDA splits the self-attention module into a short-distance one and a long-distance counterpart, which not only reduces the computational burden but also keeps both small-scale and large-scale features in the tokens. Moreover, through experiments on CrossFormer, we observe another two issues that affect vision transformers' performance, i.e., the enlarging self-attention maps and amplitude explosion. Thus, we further propose a progressive group size (PGS) paradigm and an amplitude cooling layer (ACL) to alleviate the two issues, respectively. The CrossFormer incorporating with PGS and ACL is called CrossFormer++. Extensive experiments show that CrossFormer++ outperforms the other vision transformers on image classification, object detection, instance segmentation, and semantic segmentation tasks.
Förderung
- National Natural Science Foundation of China
- Key R&D Program of Zhejiang Province, China
- Yongjiang Talent Introduction Programme
Zitieren
APA
Wang, W., Chen, W., Qiu, Q., Chen, L., Wu, B., Lin, B., He, X., & Liu, W. (2024). CrossFormer++: A Versatile Vision Transformer Hinging on Cross-Scale Attention. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(5), 3123-3136. https://doi.org/10.1109/tpami.2023.3341806
Vancouver
Wang W, Chen W, Qiu Q, Chen L, Wu B, Lin B, et al. CrossFormer++: A Versatile Vision Transformer Hinging on Cross-Scale Attention. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(5):3123-3136. doi:10.1109/tpami.2023.3341806
BibTeX
@article{Wang2024,
author = {Wang, Wenxiao and Chen, Wei and Qiu, Qibo and Chen, Long and Wu, Boxi and Lin, Binbin and He, Xiaofei and Liu, Wei},
title = {CrossFormer++: A Versatile Vision Transformer Hinging on Cross-Scale Attention},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2024},
volume = {46},
number = {5},
pages = {3123--3136},
doi = {10.1109/tpami.2023.3341806}
}
EUVIMED