Achieving Constant Memory Complexity in Long Context Transformers Through Linear Attention
Academic Journal of Applied Sciences · 19.05.2026 · 1 (3) · 106-112
DOI: 10.54097/drkv6s13 ↗Kurz gefasst
Fachartikel in der Zeitschrift Academic Journal of Applied Sciences, online erschienen am 19.05.2026. Verfasst von Zeyuan Liu und 2 weiteren Autor:innen. Die Arbeit nennt 29 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
The transformer architecture has emerged as the dominant paradigm for sequence modeling, yet its standard self-attention mechanism imposes quadratic time and memory cost with respect to sequence length, presenting a fundamental scalability barrier for long-context applications. This paper investigates linear attention as a principled mechanism for achieving constant memory complexity (KM) during autoregressive inference in transformer models. By replacing the softmax normalization in scaled dot-product attention with a kernel decomposition, the computation is restructured so that keys and values are combined before interacting with queries, yielding an equivalent recurrent form that maintains a fixed-size hidden state regardless of sequence length. We present a unified theoretical derivation of this parallel-to-recurrent equivalence, introduce a learnable positive feature map paired with data-dependent gating and structured state normalization, and evaluate the resulting architecture on long-context language modeling and downstream comprehension benchmarks. Empirical results confirm that peak GPU memory consumption remains constant at 2.1 gigabytes across context lengths from 1,024 to 131,072 tokens, achieving a 13.6× throughput advantage over standard softmax attention at length 65,536, with only a 4.8% relative perplexity increase on the Pile dataset. These findings establish constant-complexity linear attention as a viable deployment mechanism for memory-constrained long-context transformer inference.
Zitieren
APA
Liu, Z., Du, H., & Eriksson, J. (2026). Achieving Constant Memory Complexity in Long Context Transformers Through Linear Attention. Academic Journal of Applied Sciences, 1(3), 106-112. https://doi.org/10.54097/drkv6s13
Vancouver
Liu Z, Du H, Eriksson J. Achieving Constant Memory Complexity in Long Context Transformers Through Linear Attention. Academic Journal of Applied Sciences. 2026;1(3):106-112. doi:10.54097/drkv6s13
BibTeX
@article{Liu2026,
author = {Liu, Zeyuan and Du, Haoxiang and Eriksson, Jonas},
title = {Achieving Constant Memory Complexity in Long Context Transformers Through Linear Attention},
journal = {Academic Journal of Applied Sciences},
year = {2026},
volume = {1},
number = {3},
pages = {106--112},
doi = {10.54097/drkv6s13}
}
EUVIMED