EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Alias-Controlled Spectral Sparsification for Million-Token Sequence Models

Ji-Ho Koo, Ye-Jun Hwang

International Journal of Information and Communication Technology Trends · 2026

DOI: 10.66525/ijictt128 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Information and Communication Technology Trends, 2026. Verfasst von Ji-Ho Koo und 1 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The rapid evolution of large language models has driven the demand for massively extended context windows, culminating in the pursuit of million-token sequence modeling capabilities. However, scaling the self-attention mechanism to such unprecedented lengths introduces prohibitive computational and memory bottlenecks due to its inherent quadratic complexity. While various sparsification and low-rank approximation techniques have been proposed, they frequently suffer from severe information loss and structural degradation when applied to extremely long sequences. This paper introduces a novel framework termed Alias-Controlled Spectral Sparsification, which leverages principles from graph signal processing to selectively prune attention graphs in the spectral domain. By rigorously analyzing the aliasing artifacts that emerge during aggressive token downsampling, we develop a specialized anti-aliasing filter that preserves the fundamental low-frequency semantic structures while systematically attenuating high-frequency noise. Our methodology enables the compression of million-token sequences into highly tractable spectral representations without sacrificing critical contextual dependencies. Extensive theoretical analysis and empirical evaluations on long-document benchmarks demonstrate that our approach significantly outperforms existing sparse attention mechanisms in both perplexity retention and hardware efficiency. The proposed architecture achieves near-linear time complexity, offering a scalable and robust solution for next-generation sequence models processing ultra-long contexts.

Zitieren

APA

Koo, J., & Hwang, Y. (2026). Alias-Controlled Spectral Sparsification for Million-Token Sequence Models. International Journal of Information and Communication Technology Trends. https://doi.org/10.66525/ijictt128

Vancouver

Koo J, Hwang Y. Alias-Controlled Spectral Sparsification for Million-Token Sequence Models. International Journal of Information and Communication Technology Trends. 2026. doi:10.66525/ijictt128

BibTeX
@article{Koo2026,
  author = {Koo, Ji-Ho and Hwang, Ye-Jun},
  title = {Alias-Controlled Spectral Sparsification for Million-Token Sequence Models},
  journal = {International Journal of Information and Communication Technology Trends},
  year = {2026},
  doi = {10.66525/ijictt128}
}
Kostenlos starten →