Dimension-Independent Approximations on Low-Dimensional Manifolds Using Transformers
Mathematics · 05.05.2026 · 14 (9) · 1559
DOI: 10.3390/math14091559 ↗Kurz gefasst
Fachartikel in der Zeitschrift Mathematics, online erschienen am 05.05.2026. Verfasst von Ji Shi und 1 weiteren Autor:innen. Die Arbeit nennt 33 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Simons Foundation.
Abstract (Originalsprache)
Deep neural networks have been remarkably successful in high-dimensional learning and scientific computing, often succeeding where classical discretization methods fail due to the curse of dimensionality. This efficacy is often explained by their approximation properties combined with the manifold hypothesis: the idea that although data are embedded in dimension D, the effective degrees of freedom are governed by a much smaller intrinsic dimension d≪D. Under this hypothesis, data are concentrated near a low-dimensional manifold that neural networks can approximate efficiently. While the approximation theory for fully-connected ReLU networks on manifolds is well established, a comparable theory for transformer architectures, the dominant model class in modern foundation models, is still emerging. In this paper, we prove a new non-asymptotic, uniform approximation theorem for a class of single-head ReLU-transformers acting on vector inputs, where the approximation error depends only on the intrinsic dimension d rather than on the ambient dimension D. To the best of our knowledge, this is the first transformer approximation result that combines an intrinsic-dimensional rate with an ambient-dimension-independent multiplicative constant. We include a numerical experiment using a circle embedded in ambient dimensions of various sizes, showing that the observed error remains nearly unchanged as D varies, in agreement with the predicted ambient-dimension independence.
Förderung
- Simons Foundation
Zitieren
APA
Shi, J., & Labate, D. (2026). Dimension-Independent Approximations on Low-Dimensional Manifolds Using Transformers. Mathematics, 14(9), 1559. https://doi.org/10.3390/math14091559
Vancouver
Shi J, Labate D. Dimension-Independent Approximations on Low-Dimensional Manifolds Using Transformers. Mathematics. 2026;14(9):1559. doi:10.3390/math14091559
BibTeX
@article{Shi2026,
author = {Shi, Ji and Labate, Demetrio},
title = {Dimension-Independent Approximations on Low-Dimensional Manifolds Using Transformers},
journal = {Mathematics},
year = {2026},
volume = {14},
number = {9},
pages = {1559},
doi = {10.3390/math14091559}
}
EUVIMED