AToM: Adaptive Token Merging for Efficient Acceleration of Vision Transformer
IEEE Transactions on Computers · 2025-05 · 74 (5) · 1620-1633
DOI: 10.1109/tc.2025.3540638 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Computers, 2025-05. Verfasst von Jaekang Shin und 4 weiteren Autor:innen. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 43 Literaturangaben. Gefördert von: MSIT (Ministry of Science and ICT), Korea, under the ITRC (Information Technology Research Center) support program, IITP.
Abstract (Originalsprache)
Recently, Vision Transformers (ViTs) have set a new standard in computer vision (CV), showing unparalleled image processing performance. However, their substantial computational requirements hinder practical deployment, especially on resource-limited devices common in CV applications. Token merging has emerged as a solution, condensing tokens with similar features to cut computational and memory demands. Yet, existing applications on ViTs often miss the mark in token compression, with rigid merging strategies and a lack of in-depth analysis of ViT merging characteristics. To overcome these issues, this paper introduces Adaptive Token Merging (AToM), a comprehensive algorithm-architecture co-design for accelerating ViTs. The AToM algorithm employs an image-adaptive, fine-grained merging strategy, significantly boosting computational efficiency. We also optimize the merging and unmerging processes to minimize overhead, employing techniques like First-Come-First-Merge mapping and Linear Distance Calculation. On the hardware side, the AToM architecture is tailor-made to exploit the AToM algorithm's benefits, with specialized engines for efficient merge and unmerge operations. Our pipeline architecture ensures end-to-end ViT processing, minimizing latency and memory overhead from the AToM algorithm. Across various hardware platforms including CPU, EdgeGPU, and GPU, AToM achieves average end-to-end speedups of 10.9 $\boldsymbol{\times}$ × , 7.7 $\boldsymbol{\times}$ × , and 5.4 $\boldsymbol{\times}$ × , alongside energy savings of 24.9 $\boldsymbol{\times}$ × , 1.8 $\boldsymbol{\times}$ × , and 16.7 $\boldsymbol{\times}$ × . Moreover, AToM offers 1.2 $\boldsymbol{\times}$ × 1.9 $\boldsymbol{\times}$ × higher effective throughput compared to existing transformer accelerators.
Förderung
- MSIT (Ministry of Science and ICT), Korea, under the ITRC (Information Technology Research Center) support program
- IITP
Zitieren
APA
Shin, J., Kang, M., Han, Y., Park, J., & Kim, L. (2025). AToM: Adaptive Token Merging for Efficient Acceleration of Vision Transformer. IEEE Transactions on Computers, 74(5), 1620-1633. https://doi.org/10.1109/tc.2025.3540638
Vancouver
Shin J, Kang M, Han Y, Park J, Kim L. AToM: Adaptive Token Merging for Efficient Acceleration of Vision Transformer. IEEE Transactions on Computers. 2025;74(5):1620-1633. doi:10.1109/tc.2025.3540638
BibTeX
@article{Shin2025,
author = {Shin, Jaekang and Kang, Myeonggu and Han, Yunki and Park, Junyoung and Kim, Lee-Sup},
title = {AToM: Adaptive Token Merging for Efficient Acceleration of Vision Transformer},
journal = {IEEE Transactions on Computers},
year = {2025},
volume = {74},
number = {5},
pages = {1620--1633},
doi = {10.1109/tc.2025.3540638}
}
EUVIMED