An Experimental Study on Exploring Strong Lightweight Vision Transformers via Masked Image Modeling Pre-training
International Journal of Computer Vision · 2025-07 · 133 (7) · 3918-3950
DOI: 10.1007/s11263-024-02327-w ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Computer Vision, online erschienen am 13.02.2025. Verfasst von Jin Gao und 9 weiteren Autor:innen. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 178 Literaturangaben.
Abstract (Originalsprache)
Masked image modeling (MIM) pre-training for large-scale vision transformers (ViTs) has enabled promising downstream performance on top of the learned self-supervised ViT features. In this paper, we question if the extremely simple lightweight ViTs’ fine-tuning performance can also benefit from this pre-training paradigm, which is considerably less studied yet in contrast to the well-established lightweight architecture design methodology. We use an observation-analysis-solution flow for our study. We first systematically observe different behaviors among the evaluated pre-training methods with respect to the downstream fine-tuning data scales. Furthermore, we analyze the layer representation similarities and attention maps across the obtained models, which clearly show the inferior learning of MIM pre-training on higher layers, leading to unsatisfactory transfer performance on data-insufficient downstream tasks. This finding is naturally a guide to designing our distillation strategies during pre-training to solve the above deterioration problem. Extensive experiments have demonstrated the effectiveness of our approach. Our pre-training with distillation on pure lightweight ViTs with vanilla/hierarchical design (5.7M/6.5M) can achieve 79.4%\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$79.4\%$$\end{document}/78.9%\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$78.9\%$$\end{document} top-1 accuracy on ImageNet-1K. It also enables SOTA performance on the ADE20K segmentation task (42.8%\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$42.8\%$$\end{document} mIoU) and LaSOT tracking task (66.1%\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$66.1\%$$\end{document} AUC) in the lightweight regime. The latter even surpasses all the current SOTA lightweight CPU-realtime trackers.
Zitieren
APA
Gao, J., Lin, S., Wang, S., Kou, Y., Li, Z., Li, L., Zhang, C., Zhang, X., Wang, Y., & Hu, W. (2025). An Experimental Study on Exploring Strong Lightweight Vision Transformers via Masked Image Modeling Pre-training. International Journal of Computer Vision, 133(7), 3918-3950. https://doi.org/10.1007/s11263-024-02327-w
Vancouver
Gao J, Lin S, Wang S, Kou Y, Li Z, Li L, et al. An Experimental Study on Exploring Strong Lightweight Vision Transformers via Masked Image Modeling Pre-training. International Journal of Computer Vision. 2025;133(7):3918-3950. doi:10.1007/s11263-024-02327-w
BibTeX
@article{Gao2025,
author = {Gao, Jin and Lin, Shubo and Wang, Shaoru and Kou, Yutong and Li, Zeming and Li, Liang and Zhang, Congxuan and Zhang, Xiaoqin and Wang, Yizheng and Hu, Weiming},
title = {An Experimental Study on Exploring Strong Lightweight Vision Transformers via Masked Image Modeling Pre-training},
journal = {International Journal of Computer Vision},
year = {2025},
volume = {133},
number = {7},
pages = {3918--3950},
doi = {10.1007/s11263-024-02327-w}
}
EUVIMED