Sequence Modeling Architectures: Foundations [Special Issue on the Mathematics of Deep Learning]
IEEE Signal Processing Magazine · 2026-05 · 43 (3) · 115-129
DOI: 10.1109/msp.2026.3682808 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Signal Processing Magazine, 2026-05. Verfasst von Felix Sarnthein und 8 weiteren Autor:innen. Die Arbeit nennt 50 Literaturangaben. Gefördert von: Swiss National Science Foundation.
Abstract (Originalsprache)
With the success of deep learning in the past decade, neural networks transitioned from simple layer stacking to sophisticated, highly complex designs. This article provides an overview of such deep learning architectures for sequence modeling, with a focus on their core mathematical principles: 1) we review the evolution of common sequence models from recurrent neural networks (RNNs), to Transformers, and modern state-space models (SSMs), exploring their foundational components such as gating, attention, and linear dynamical systems mechanisms. 2) We present general linear sequence mixers as a unifying framework to highlight the relationship between architectures and compare them in terms of inductive biases and algorithmic complexity. 3) We examine how practical considerations such as parallelizability and hardware awareness dictate their success. Finally, we discuss their impact on diverse fields of signal processing, including time-series analysis, natural language, and computer vision.
Förderung
- Swiss National Science Foundation
Zitieren
APA
Sarnthein, F., Afzal, A., Pascanu, R., Orvieto, A., Gulcehre, C., Geiping, J., Tonin, F., Gu, A., & Cevher, V. (2026). Sequence Modeling Architectures: Foundations [Special Issue on the Mathematics of Deep Learning]. IEEE Signal Processing Magazine, 43(3), 115-129. https://doi.org/10.1109/msp.2026.3682808
Vancouver
Sarnthein F, Afzal A, Pascanu R, Orvieto A, Gulcehre C, Geiping J, et al. Sequence Modeling Architectures: Foundations [Special Issue on the Mathematics of Deep Learning]. IEEE Signal Processing Magazine. 2026;43(3):115-129. doi:10.1109/msp.2026.3682808
BibTeX
@article{Sarnthein2026,
author = {Sarnthein, Felix and Afzal, Arshia and Pascanu, Razvan and Orvieto, Antonio and Gulcehre, Caglar and Geiping, Jonas and Tonin, Francesco and Gu, Albert and Cevher, Volkan},
title = {Sequence Modeling Architectures: Foundations [Special Issue on the Mathematics of Deep Learning]},
journal = {IEEE Signal Processing Magazine},
year = {2026},
volume = {43},
number = {3},
pages = {115--129},
doi = {10.1109/msp.2026.3682808}
}
EUVIMED