BinaryViT: Toward Efficient and Accurate Binary Vision Transformers
IEEE Transactions on Circuits and Systems for Video Technology · 2025-01 · 35 (1) · 195-206
DOI: 10.1109/tcsvt.2024.3457610 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems for Video Technology, 2025-01. Verfasst von Junrui Xiao und 4 weiteren Autor:innen. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 75 Literaturangaben. Gefördert von: National Science and Technology Major Project of China, National Natural Science Foundation of China.
Abstract (Originalsprache)
Vision Transformers (ViTs) have emerged as the new fundamental architecture for most computer vision fields. However, the considerable memory and computation costs also hinder their application on resource-limited devices. Currently, binarization has demonstrated remarkable potential as a model compression technique in traditional Convolutional Neural Networks (CNNs), albeit with some accuracy loss. In this paper, we focus on binarization of ViTs, which is still under-studied and suffering a significant performance drop. We start with constructing a strong baseline of binary ViTs, integrating some of the best practices from binary CNNs, which forms the foundation of our exploration. Subsequently, we identify that the severe performance degradation of the baseline is mainly caused by the weight oscillation around the quantization boundary and the information distortion in the activation of ViTs. To address these challenges, we introduce BinaryViT, a precise full binarization framework tailored for Vision Transformers (ViTs), effectively pushing the binarization of ViTs to its limit. Specifically, we propose a novel gradient regularization scheme (GRS), which mitigates oscillations by fostering a smooth moving of latent weights to be away from the quantization boundary during the training process. Additionally, we have devised an Activation Shift Module (ASM) that dynamically adjusts the activation distribution prior to the sign function, thereby minimizing the information distortion stemming from the significant inter-channel variations. Extensive experiments on ImageNet dataset show that our BinaryViT consistently surpasses the strong baseline by 2.05% and improves the accuracy of fully binarized ViTs to a usable level. Furthermore, our method achieves impressive savings of $16.2\times $ and $17.7\times $ in model size and OPs compared to the full-precision DeiT-S.
Förderung
- National Science and Technology Major Project of China
- National Natural Science Foundation of China
Zitieren
APA
Xiao, J., Li, Z., Li, J., Yang, L., & Gu, Q. (2025). BinaryViT: Toward Efficient and Accurate Binary Vision Transformers. IEEE Transactions on Circuits and Systems for Video Technology, 35(1), 195-206. https://doi.org/10.1109/tcsvt.2024.3457610
Vancouver
Xiao J, Li Z, Li J, Yang L, Gu Q. BinaryViT: Toward Efficient and Accurate Binary Vision Transformers. IEEE Transactions on Circuits and Systems for Video Technology. 2025;35(1):195-206. doi:10.1109/tcsvt.2024.3457610
BibTeX
@article{Xiao2025,
author = {Xiao, Junrui and Li, Zhikai and Li, Jianquan and Yang, Lianwei and Gu, Qingyi},
title = {BinaryViT: Toward Efficient and Accurate Binary Vision Transformers},
journal = {IEEE Transactions on Circuits and Systems for Video Technology},
year = {2025},
volume = {35},
number = {1},
pages = {195--206},
doi = {10.1109/tcsvt.2024.3457610}
}
EUVIMED