EUVIMEDEUVIMED entdecken
Crossref

Why Self-Attention Is Natural for Sequence-to-Sequence Problems? A Perspective from Symmetries

Chao Ma, Lexing Ying

Journal of Machine Learning · 2023

DOI: 10.4208/jml.221206 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Machine Learning, 2023. Verfasst von Chao Ma und 1 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 58 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In this paper, we show that structures similar to self-attention are natural for learning many sequence-to-sequence problems from the perspective of symmetry. Inspired by language processing applications, we study the orthogonal equivariance of seq2seq functions with knowledge, which are functions taking two inputs — an input sequence and a knowledge — and outputting another sequence. The knowledge consists of a set of vectors in the same embedding space as the input sequence, containing the information of the language used to process the input sequence. We show that orthogonal equivariance in the embedding space is natural for seq2seq functions with knowledge, and under such equivariance, the function must take a form close to self-attention. This shows that network structures similar to self-attention are the right structures for representing the target function of many seq2seq problems. The representation can be further refined if a finite information principle is considered, or a permutation equivariance holds for the elements of the input sequence.

Zitieren

APA

Ma, C., & Ying, L. (2023). Why Self-Attention Is Natural for Sequence-to-Sequence Problems? A Perspective from Symmetries. Journal of Machine Learning. https://doi.org/10.4208/jml.221206

Vancouver

Ma C, Ying L. Why Self-Attention Is Natural for Sequence-to-Sequence Problems? A Perspective from Symmetries. Journal of Machine Learning. 2023. doi:10.4208/jml.221206

BibTeX
@article{Ma2023,
  author = {Ma, Chao and Ying, Lexing},
  title = {Why Self-Attention Is Natural for Sequence-to-Sequence Problems? A Perspective from Symmetries},
  journal = {Journal of Machine Learning},
  year = {2023},
  doi = {10.4208/jml.221206}
}
Kostenlos starten →