EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Value Matrix Dynamics and Attractor Selection in a Reduced Transformer Model

Vitaly Volpert

Mathematics · 15.09.2026 · 14 (18) · 3350

DOI: 10.3390/math14183350 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Mathematics, online erschienen am 15.09.2026. Verfasst von Vitaly Volpert. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 16 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: state assignment.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We study an interacting particle system on the unit sphere Sd−1 motivated by the forward-pass dynamics of a transformer model that combines a value matrix V and a u-aligned multi-layer perceptron (MLP) Φ, a particular form of MLP specified below. For a symmetric value matrix and this MLP, the one-cluster dynamic is the Riemannian gradient flow of an explicit Lyapunov function L=fV+g, which combines the Rayleigh quotient of V with the MLP potential. Every trajectory converges to a fixed point, limit cycles are excluded, and the fixed points are the critical points of L: local maxima are stable attractors and saddles are unstable. Isolating the value matrix (Φ≡0, with V possibly non-symmetric and a real strictly dominant eigenvalue λ1), we show that the dominant eigenvector v1 is the selected attractor and that its basin is almost the whole sphere, the two basins of ±v1 being separated by an invariant subsphere carried by the left eigenvector of V. The subdominant eigenvectors form a hierarchical structure of saddles, and a complex dominant pair replaces the point attractor by a great-circle limit cycle. When the tokens move independently, softmax attention contributes no first-order clustering force: the tokens collapse to ±v1 at a consensus rate equal to λ1 and independent of the attention matrix, and whether two clusters in opposite basins coexist or merge is governed by the single order parameter s=v1⊤Av1. Thus, the value matrix, rather than the attention, controls the location and the selection of the limiting cluster. All results are confirmed by numerical integration of the continuous-time ordinary differential equation (ODE).

Förderung

  • state assignment

Zitieren

APA

Volpert, V. (2026). Value Matrix Dynamics and Attractor Selection in a Reduced Transformer Model. Mathematics, 14(18), 3350. https://doi.org/10.3390/math14183350

Vancouver

Volpert V. Value Matrix Dynamics and Attractor Selection in a Reduced Transformer Model. Mathematics. 2026;14(18):3350. doi:10.3390/math14183350

BibTeX
@article{Volpert2026,
  author = {Volpert, Vitaly},
  title = {Value Matrix Dynamics and Attractor Selection in a Reduced Transformer Model},
  journal = {Mathematics},
  year = {2026},
  volume = {14},
  number = {18},
  pages = {3350},
  doi = {10.3390/math14183350}
}
Kostenlos starten →