On the Convergence of Gradient Descent on Learning Transformers With Residual Connections
IEEE Signal Processing Letters · 2026 · 33 · 1661-1665
DOI: 10.1109/lsp.2026.3684818 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Signal Processing Letters, 2026. Verfasst von Zhen Qin und 3 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 38 Literaturangaben. Gefördert von: NSF.
Abstract (Originalsprache)
Transformer models have emerged as fundamental tools across various scientific and engineering disciplines, owing to their outstanding performance in diverse applications. Despite this empirical success, the theoretical foundations of Transformers remain relatively underdeveloped, particularly in understanding their training dynamics. Existing research predominantly examines isolated components–such as self-attention mechanisms and feedforward networks–without thoroughly investigating the interdependencies between these components, especially when residual connections are present. In this paper, we aim to bridge this gap by analyzing the convergence behavior of a structurally complete yet single-layer Transformer, comprising self-attention, a feedforward network, and residual connections. We demonstrate that, under appropriate initialization, gradient descent exhibits a linear convergence rate, where the convergence speed is determined by the minimum and maximum singular values of the output matrix from the attention layer. Moreover, our convergence analysis establishes a theoretical characterization of residual connections by showing that they alleviate the ill-conditioning of the attention output matrix, which arises from the low-rank structure induced by the softmax operation, thereby improving optimization stability. Empirical results corroborate our theoretical insights, illustrating the beneficial role of residual connections in promoting convergence stability.
Förderung
- NSF
Zitieren
APA
Qin, Z., Zhou, J., Jiang, J., & Zhu, Z. (2026). On the Convergence of Gradient Descent on Learning Transformers With Residual Connections. IEEE Signal Processing Letters, 33, 1661-1665. https://doi.org/10.1109/lsp.2026.3684818
Vancouver
Qin Z, Zhou J, Jiang J, Zhu Z. On the Convergence of Gradient Descent on Learning Transformers With Residual Connections. IEEE Signal Processing Letters. 2026;33:1661-1665. doi:10.1109/lsp.2026.3684818
BibTeX
@article{Qin2026,
author = {Qin, Zhen and Zhou, Jinxin and Jiang, Jiachen and Zhu, Zhihui},
title = {On the Convergence of Gradient Descent on Learning Transformers With Residual Connections},
journal = {IEEE Signal Processing Letters},
year = {2026},
volume = {33},
pages = {1661--1665},
doi = {10.1109/lsp.2026.3684818}
}
EUVIMED