Why Self-Attention Is Natural for Sequence-to-Sequence Problems? A Perspective from Symmetries
Journal of Machine Learning · 2023
DOI: 10.4208/jml.221206 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Machine Learning, 2023. Verfasst von Chao Ma und 1 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 58 Literaturangaben.
Abstract (Originalsprache)
In this paper, we show that structures similar to self-attention are natural for learning many sequence-to-sequence problems from the perspective of symmetry. Inspired by language processing applications, we study the orthogonal equivariance of seq2seq functions with knowledge, which are functions taking two inputs — an input sequence and a knowledge — and outputting another sequence. The knowledge consists of a set of vectors in the same embedding space as the input sequence, containing the information of the language used to process the input sequence. We show that orthogonal equivariance in the embedding space is natural for seq2seq functions with knowledge, and under such equivariance, the function must take a form close to self-attention. This shows that network structures similar to self-attention are the right structures for representing the target function of many seq2seq problems. The representation can be further refined if a finite information principle is considered, or a permutation equivariance holds for the elements of the input sequence.
Zitieren
APA
Ma, C., & Ying, L. (2023). Why Self-Attention Is Natural for Sequence-to-Sequence Problems? A Perspective from Symmetries. Journal of Machine Learning. https://doi.org/10.4208/jml.221206
Vancouver
Ma C, Ying L. Why Self-Attention Is Natural for Sequence-to-Sequence Problems? A Perspective from Symmetries. Journal of Machine Learning. 2023. doi:10.4208/jml.221206
BibTeX
@article{Ma2023,
author = {Ma, Chao and Ying, Lexing},
title = {Why Self-Attention Is Natural for Sequence-to-Sequence Problems? A Perspective from Symmetries},
journal = {Journal of Machine Learning},
year = {2023},
doi = {10.4208/jml.221206}
}
EUVIMED