EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Sparse Projection Attention: A Computationally Efficient Framework for Long Sequence Modeling

Mehdi Chrifi Alaoui, Nour-Eddine Joudar, Mohamed Ettaouil

Mathematics · 23.05.2026 · 14 (11) · 1813

DOI: 10.3390/math14111813 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Mathematics, online erschienen am 23.05.2026. Verfasst von Mehdi Chrifi Alaoui und 2 weiteren Autor:innen. Die Arbeit nennt 33 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The self-attention mechanism has revolutionized sequence modeling but suffers from quadratic computational complexity with respect to sequence length, limiting its applicability to long sequences. We propose Sparse Projection Attention (SPA), a novel attention variant that leverages learnable sparse projections to reduce the effective dimensionality of queries and keys while maintaining expressive power. Our method is grounded in the Johnson–Lindenstrauss lemma and provides theoretical guarantees on distance preservation for fixed random projection variants. We introduce a comprehensive mathematical framework including error bounds, convergence analysis, and gradient dynamics. Experimental results on language modeling, machine translation, and long-range sequence classification demonstrate that SPA achieves up to 8× speedup in attention score computation, and approximately 2× end-to-end speedup, while maintaining competitive performance compared to standard attention and other efficient variants. The proposed approach offers an effective trade-off between computational efficiency and model expressivity for long-sequence tasks, making transformers more accessible for resource-constrained environments and real-time applications.

Zitieren

APA

Alaoui, M. C., Joudar, N., & Ettaouil, M. (2026). Sparse Projection Attention: A Computationally Efficient Framework for Long Sequence Modeling. Mathematics, 14(11), 1813. https://doi.org/10.3390/math14111813

Vancouver

Alaoui MC, Joudar N, Ettaouil M. Sparse Projection Attention: A Computationally Efficient Framework for Long Sequence Modeling. Mathematics. 2026;14(11):1813. doi:10.3390/math14111813

BibTeX
@article{Alaoui2026,
  author = {Alaoui, Mehdi Chrifi and Joudar, Nour-Eddine and Ettaouil, Mohamed},
  title = {Sparse Projection Attention: A Computationally Efficient Framework for Long Sequence Modeling},
  journal = {Mathematics},
  year = {2026},
  volume = {14},
  number = {11},
  pages = {1813},
  doi = {10.3390/math14111813}
}
Kostenlos starten →