EUVIMEDEUVIMED entdecken
Crossref

A Mathematical Explanation of Transformers

Xue-Cheng Tai, Hao Liu, Lingfeng Li, Raymond H. Chan

SIAM Journal on Imaging Sciences · 30.09.2026 · 19 (3) · 1542-1568

DOI: 10.1137/25m1768497 ↗

Kurz gefasst

Fachartikel in der Zeitschrift SIAM Journal on Imaging Sciences, online erschienen am 17.07.2026. Verfasst von Xue-Cheng Tai und 3 weiteren Autor:innen. Die Arbeit nennt 66 Literaturangaben. Gefördert von: NORCE, HKRGC, Hetao Institute of Mathematics and Interdisciplinary Sciences.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The Transformer architecture has revolutionized the field of sequence modeling and underpins the recent breakthroughs in large language models (LLMs). However, a comprehensive mathematical theory that explains its structure and operations remains elusive. In this work, we propose a novel continuous framework that rigorously interprets the Transformer as a discretization of a structured integro-differential equation. Within this formulation, the self-attention mechanism emerges naturally as a nonlocal integral operator, and layer normalization is characterized as a projection to a time-dependent constraint. This operator-theoretic and variational perspective offers a unified and interpretable foundation for understanding the architecture’s core components, including attention, feedforward layers, and normalization. Our approach extends beyond previous theoretical analyses by embedding the entire Transformer operation in continuous domains for both token indices and feature dimensions. This leads to a principled and flexible framework that not only deepens on theoretical insight but also offers new directions for architecture design, analysis, and control-based interpretations. This new interpretation provides a step toward bridging the gap between deep learning architectures and continuous mathematical modeling, and contributes a foundational perspective to the ongoing development of interpretable and theoretically grounded neural network models.

Förderung

  • NORCE
  • HKRGC
  • Hetao Institute of Mathematics and Interdisciplinary Sciences
  • ITF

Zitieren

APA

Tai, X., Liu, H., Li, L., & Chan, R. H. (2026). A Mathematical Explanation of Transformers. SIAM Journal on Imaging Sciences, 19(3), 1542-1568. https://doi.org/10.1137/25m1768497

Vancouver

Tai X, Liu H, Li L, Chan RH. A Mathematical Explanation of Transformers. SIAM Journal on Imaging Sciences. 2026;19(3):1542-1568. doi:10.1137/25m1768497

BibTeX
@article{Tai2026,
  author = {Tai, Xue-Cheng and Liu, Hao and Li, Lingfeng and Chan, Raymond H.},
  title = {A Mathematical Explanation of Transformers},
  journal = {SIAM Journal on Imaging Sciences},
  year = {2026},
  volume = {19},
  number = {3},
  pages = {1542--1568},
  doi = {10.1137/25m1768497}
}
Kostenlos starten →