Heat-kernel context compression for transformer representations using graph diffusion
Academia AI and Applications · 06.11.2026 · 2 (4)
DOI: 10.20935/acadai8620 ↗Kurz gefasst
Fachartikel in der Zeitschrift Academia AI and Applications, online erschienen am 06.11.2026. Verfasst von Shashank Reddy Vadyala und 1 weiteren Autor:innen. Die Arbeit nennt 16 Literaturangaben.
Abstract (Originalsprache)
Introduction
Long contexts often contain geometrically redundant internal representations. Heat Kernel Context Compression (HKCC) is a training-free representation–compression method for transformer systems in which internal embeddings and attention logits can be modified. Materials and methods: Context representations are placed on a sparse similarity graph. The method uses the random-walk Laplacian throughout, while symmetric normalization appears only through an explicit similarity transform used for stable Chebyshev computation. Heat residuals select anchors, and omitted representations are merged into multiplicity-weighted representatives.
Results
Corrected simulations with 340 unit-normalized vectors show that the random-walk heat kernel preserves row sums exactly and agrees with its similarity-transformed symmetric implementation to numerical precision. Across six controlled contexts, multiplicity-aware merging preserved approximately 0.99999 attention output cosine fidelity at 8% and 12% retention, whereas residual-only pruning degraded sharply. Anchor rules were effectively tied under this synthetic regime.
Conclusions
HKCC is supported as a controlled proof of concept for representation–compression in self-hosted or otherwise modifiable transformer stacks. The current study does not establish savings for commercial text application programming interfaces (APIs), downstream task preservation, or universal superiority of heat residual anchors. The added utility of diffusion, real-model key/value and positional integration, and end-to-end prefill speedups remain unestablished.
Zitieren
APA
Vadyala, S. R., & Betgeri, S. N. (2026). Heat-kernel context compression for transformer representations using graph diffusion. Academia AI and Applications, 2(4). https://doi.org/10.20935/acadai8620
Vancouver
Vadyala SR, Betgeri SN. Heat-kernel context compression for transformer representations using graph diffusion. Academia AI and Applications. 2026;2(4). doi:10.20935/acadai8620
BibTeX
@article{Vadyala2026,
author = {Vadyala, Shashank Reddy and Betgeri, Sai Nethra},
title = {Heat-kernel context compression for transformer representations using graph diffusion},
journal = {Academia AI and Applications},
year = {2026},
volume = {2},
number = {4},
doi = {10.20935/acadai8620}
}
EUVIMED