Hybrid Attention Spike Transformer
IET Cyber-Systems and Robotics · 2025-01 · 7 (1)
DOI: 10.1049/csy2.70010 ↗Kurz gefasst
Fachartikel in der Zeitschrift IET Cyber-Systems and Robotics, online erschienen am 10.04.2025. Verfasst von Xiongfei Fan und 2 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 47 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
Spike transformers cannot be pretrained due to objective factors such as lack of datasets and memory constraints, which results in a significant performance gap compared to pretrained artificial neural networks (ANNs), thereby hindering their practical applicability. To address this issue, we propose a hybrid attention spike transformer that utilises self‐attention with compound tokens and channel attention‐based token processing to better capture the inductive biases of the data. We also add convolution in patch splitting and feedforward networks, which not only provides local information but also leverages the translation invariance and locality of convolutions to help the model converge. Experiments on static datasets and neuromorphic datasets demonstrate that our method achieves state‐of‐the‐art performance in the spiking neural networks (SNNs) field. Notably, we achieve a top‐1 accuracy of 80.59% on CIFAR‐100 with only 4 time steps. As far as we know, it is the first exploration of the spike transformer with multiattention fusion, achieving outstanding effectiveness.
Zitieren
APA
Fan, X., Zhang, H., & Zhang, Y. (2025). Hybrid Attention Spike Transformer. IET Cyber-Systems and Robotics, 7(1). https://doi.org/10.1049/csy2.70010
Vancouver
Fan X, Zhang H, Zhang Y. Hybrid Attention Spike Transformer. IET Cyber-Systems and Robotics. 2025;7(1). doi:10.1049/csy2.70010
BibTeX
@article{Fan2025,
author = {Fan, Xiongfei and Zhang, Hong and Zhang, Yu},
title = {Hybrid Attention Spike Transformer},
journal = {IET Cyber-Systems and Robotics},
year = {2025},
volume = {7},
number = {1},
doi = {10.1049/csy2.70010}
}
EUVIMED