Demystify Transformers & Convolutions in Modern Image Deep Networks
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-04 · 47 (4) · 2416-2428
DOI: 10.1109/tpami.2024.3520508 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-04. Verfasst von Xiaowei Hu und 11 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 54 Literaturangaben. Gefördert von: National Key R&D Program of China, National Natural Science Foundation of China.
Abstract (Originalsprache)
Vision transformers have gained popularity recently, leading to the development of new vision backbones with improved features and consistent performance gains. However, these advancements are not solely attributable to novel feature transformation designs; certain benefits also arise from advanced network-level and block-level architectures. This paper aims to identify the real gains of popular convolution and attention operators through a detailed study. We find that the key difference among these feature transformation modules, such as attention or convolution, lies in their spatial feature aggregation approach, known as the "spatial token mixer" (STM). To facilitate an impartial comparison, we introduce a unified architecture to neutralize the impact of divergent network-level and block-level designs. Subsequently, various STMs are integrated into this unified framework for comprehensive comparative analysis. Our experiments on various tasks and an analysis of inductive bias show a significant performance boost due to advanced network-level and block-level designs, but performance differences persist among different STMs. Our detailed analysis also reveals various findings about different STMs, including effective receptive fields, invariance, and adversarial robustness tests.
Förderung
- National Key R&D Program of China
- National Natural Science Foundation of China
Zitieren
APA
Hu, X., Shi, M., Wang, W., Wu, S., Xing, L., Wang, W., Zhou, X., Lu, L., Zhou, J., Wang, X., Qiao, Y., & Dai, J. (2025). Demystify Transformers & Convolutions in Modern Image Deep Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(4), 2416-2428. https://doi.org/10.1109/tpami.2024.3520508
Vancouver
Hu X, Shi M, Wang W, Wu S, Xing L, Wang W, et al. Demystify Transformers & Convolutions in Modern Image Deep Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(4):2416-2428. doi:10.1109/tpami.2024.3520508
BibTeX
@article{Hu2025,
author = {Hu, Xiaowei and Shi, Min and Wang, Weiyun and Wu, Sitong and Xing, Linjie and Wang, Wenhai and Zhou, Xizhou and Lu, Lewei and Zhou, Jie and Wang, Xiaogang and Qiao, Yu and Dai, Jifeng},
title = {Demystify Transformers & Convolutions in Modern Image Deep Networks},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2025},
volume = {47},
number = {4},
pages = {2416--2428},
doi = {10.1109/tpami.2024.3520508}
}
EUVIMED