Value Matrix Dynamics and Attractor Selection in a Reduced Transformer Model
Mathematics · 15.09.2026 · 14 (18) · 3350
DOI: 10.3390/math14183350 ↗Kurz gefasst
Fachartikel in der Zeitschrift Mathematics, online erschienen am 15.09.2026. Verfasst von Vitaly Volpert. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 16 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: state assignment.
Abstract (Originalsprache)
We study an interacting particle system on the unit sphere Sd−1 motivated by the forward-pass dynamics of a transformer model that combines a value matrix V and a u-aligned multi-layer perceptron (MLP) Φ, a particular form of MLP specified below. For a symmetric value matrix and this MLP, the one-cluster dynamic is the Riemannian gradient flow of an explicit Lyapunov function L=fV+g, which combines the Rayleigh quotient of V with the MLP potential. Every trajectory converges to a fixed point, limit cycles are excluded, and the fixed points are the critical points of L: local maxima are stable attractors and saddles are unstable. Isolating the value matrix (Φ≡0, with V possibly non-symmetric and a real strictly dominant eigenvalue λ1), we show that the dominant eigenvector v1 is the selected attractor and that its basin is almost the whole sphere, the two basins of ±v1 being separated by an invariant subsphere carried by the left eigenvector of V. The subdominant eigenvectors form a hierarchical structure of saddles, and a complex dominant pair replaces the point attractor by a great-circle limit cycle. When the tokens move independently, softmax attention contributes no first-order clustering force: the tokens collapse to ±v1 at a consensus rate equal to λ1 and independent of the attention matrix, and whether two clusters in opposite basins coexist or merge is governed by the single order parameter s=v1⊤Av1. Thus, the value matrix, rather than the attention, controls the location and the selection of the limiting cluster. All results are confirmed by numerical integration of the continuous-time ordinary differential equation (ODE).
Förderung
- state assignment
Zitieren
APA
Volpert, V. (2026). Value Matrix Dynamics and Attractor Selection in a Reduced Transformer Model. Mathematics, 14(18), 3350. https://doi.org/10.3390/math14183350
Vancouver
Volpert V. Value Matrix Dynamics and Attractor Selection in a Reduced Transformer Model. Mathematics. 2026;14(18):3350. doi:10.3390/math14183350
BibTeX
@article{Volpert2026,
author = {Volpert, Vitaly},
title = {Value Matrix Dynamics and Attractor Selection in a Reduced Transformer Model},
journal = {Mathematics},
year = {2026},
volume = {14},
number = {18},
pages = {3350},
doi = {10.3390/math14183350}
}
EUVIMED