EUVIMEDEUVIMED entdecken
Crossref

Sequence Modeling Architectures: Foundations [Special Issue on the Mathematics of Deep Learning]

Felix Sarnthein, Arshia Afzal, Razvan Pascanu, Antonio Orvieto, Caglar Gulcehre, Jonas Geiping, Francesco Tonin, Albert Gu, Volkan Cevher

IEEE Signal Processing Magazine · 2026-05 · 43 (3) · 115-129

DOI: 10.1109/msp.2026.3682808 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Signal Processing Magazine, 2026-05. Verfasst von Felix Sarnthein und 8 weiteren Autor:innen. Die Arbeit nennt 50 Literaturangaben. Gefördert von: Swiss National Science Foundation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

With the success of deep learning in the past decade, neural networks transitioned from simple layer stacking to sophisticated, highly complex designs. This article provides an overview of such deep learning architectures for sequence modeling, with a focus on their core mathematical principles: 1) we review the evolution of common sequence models from recurrent neural networks (RNNs), to Transformers, and modern state-space models (SSMs), exploring their foundational components such as gating, attention, and linear dynamical systems mechanisms. 2) We present general linear sequence mixers as a unifying framework to highlight the relationship between architectures and compare them in terms of inductive biases and algorithmic complexity. 3) We examine how practical considerations such as parallelizability and hardware awareness dictate their success. Finally, we discuss their impact on diverse fields of signal processing, including time-series analysis, natural language, and computer vision.

Förderung

  • Swiss National Science Foundation

Zitieren

APA

Sarnthein, F., Afzal, A., Pascanu, R., Orvieto, A., Gulcehre, C., Geiping, J., Tonin, F., Gu, A., & Cevher, V. (2026). Sequence Modeling Architectures: Foundations [Special Issue on the Mathematics of Deep Learning]. IEEE Signal Processing Magazine, 43(3), 115-129. https://doi.org/10.1109/msp.2026.3682808

Vancouver

Sarnthein F, Afzal A, Pascanu R, Orvieto A, Gulcehre C, Geiping J, et al. Sequence Modeling Architectures: Foundations [Special Issue on the Mathematics of Deep Learning]. IEEE Signal Processing Magazine. 2026;43(3):115-129. doi:10.1109/msp.2026.3682808

BibTeX
@article{Sarnthein2026,
  author = {Sarnthein, Felix and Afzal, Arshia and Pascanu, Razvan and Orvieto, Antonio and Gulcehre, Caglar and Geiping, Jonas and Tonin, Francesco and Gu, Albert and Cevher, Volkan},
  title = {Sequence Modeling Architectures: Foundations [Special Issue on the Mathematics of Deep Learning]},
  journal = {IEEE Signal Processing Magazine},
  year = {2026},
  volume = {43},
  number = {3},
  pages = {115--129},
  doi = {10.1109/msp.2026.3682808}
}
Kostenlos starten →