EUVIMEDEUVIMED entdecken
Crossref

Transformer Models for Signal Processing: Scaled Dot-Product Attention Implements Constrained Filtering

Terence D Sanger

Neural Computation · 22.09.2025 · 37 (10) · 1839-1852

DOI: 10.1162/neco.a.29 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Neural Computation, online erschienen am 22.09.2025. Verfasst von Terence D Sanger. Beteiligte Einrichtung: Department of Electrical Engineering, University of California, Irvine, Irvine, CA 92697, USA. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 23 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The remarkable success of the transformer machine learning architecture for processing language sequences far exceeds the performance of classical signal processing methods. A unique component of transformer models is the scaled dot-product attention (SDPA) layer, which does not appear to have an analog in prior signal processing algorithms. Here, we show that SDPA operates using a novel principle that projects the current state estimate onto the space spanned by prior estimates. We show that SDPA, when used for causal recursive state estimation, implements constrained state estimation in circumstances where the constraint is unknown and may be time varying. Since constraints in high-dimensional space may represent the complex relationships that define nonlinear signals and models, this suggests that the SDPA layer and transformer models leverage constrained estimation to achieve their success. This also suggests that transformers and the SPDA layer could be a computational model for previously unexplained capabilities of human behavior.

Autor:innen und Institutionen

  • Department of Electrical Engineering, University of California, Irvine, Irvine, CA 92697, USA.
  • Children's Hospital of Orange County, Orange, CA 92868, USA terry@sangerlab.net.

Zitieren

APA

Sanger, T. D. (2025). Transformer Models for Signal Processing: Scaled Dot-Product Attention Implements Constrained Filtering. Neural Computation, 37(10), 1839-1852. https://doi.org/10.1162/neco.a.29

Vancouver

Sanger TD. Transformer Models for Signal Processing: Scaled Dot-Product Attention Implements Constrained Filtering. Neural Computation. 2025;37(10):1839-1852. doi:10.1162/neco.a.29

BibTeX
@article{Sanger2025,
  author = {Sanger, Terence D},
  title = {Transformer Models for Signal Processing: Scaled Dot-Product Attention Implements Constrained Filtering},
  journal = {Neural Computation},
  year = {2025},
  volume = {37},
  number = {10},
  pages = {1839--1852},
  doi = {10.1162/neco.a.29}
}
Kostenlos starten →