EUVIMEDEUVIMED entdecken
Crossref

Cumulant Attention in Vision Transformers (Student Abstract)

Yuto Morimoto, Zhipeng Wang, Koji Yasuda

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (48) · 41331-41333

DOI: 10.1609/aaai.v40i48.42259 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Yuto Morimoto und 2 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Transformer models have achieved remarkable success across diverse deep learning fields, including natural language processing (NLP) and computer vision (CV). One drawback of these models is that the computational cost of the softmax attention, the core component of the transformer, exhibits quadratic complexity in both time and memory. As data scales up various attempts have been reported to overcome this bottleneck. The objective of this study is to propose a novel attention mechanism, "Cumulant Attention", that systematically balances efficiency and accuracy. This proposal introduces a statistical-mechanics perspective and a reliable approximation based on cumulant expansion into the attention layer. The low-order variant reduces computational complexity to linear order, similar to the linear attention, while keeping nonlinearity of the softmax attention. We evaluate several variants on CV tasks, including image classification with ViT on ImageNet-100 and video classification with ViViT on UCF-101. Experimental results demonstrate that the cumulant attention outperforms the linear attention and achieves accuracy comparable to the softmax attention. These findings validate the effectiveness of our approach and highlight future directions, including scaling to larger models, extending to other modalities, and optimizing implementations for GPU hardware.

Zitieren

APA

Morimoto, Y., Wang, Z., & Yasuda, K. (2026). Cumulant Attention in Vision Transformers (Student Abstract). Proceedings of the AAAI Conference on Artificial Intelligence, 40(48), 41331-41333. https://doi.org/10.1609/aaai.v40i48.42259

Vancouver

Morimoto Y, Wang Z, Yasuda K. Cumulant Attention in Vision Transformers (Student Abstract). Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(48):41331-41333. doi:10.1609/aaai.v40i48.42259

BibTeX
@article{Morimoto2026,
  author = {Morimoto, Yuto and Wang, Zhipeng and Yasuda, Koji},
  title = {Cumulant Attention in Vision Transformers (Student Abstract)},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {48},
  pages = {41331--41333},
  doi = {10.1609/aaai.v40i48.42259}
}
Kostenlos starten →