EUVIMEDEUVIMED entdecken
Crossref

Sparse Attention Across Multiple-Context KV Cache

Ziyi Cao, Qingyi Si, Jingbin Zhang, Bingquan Liu

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (36) · 30165-30173

DOI: 10.1609/aaai.v40i36.40266 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Ziyi Cao und 3 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models face significant cost challenges in long-sequence inference. To address this, reusing historical Key-Value (KV) Cache for improved inference efficiency has become a mainstream approach. Recent advances further enhance throughput by sparse attention mechanisms to select the most relevant KV Cache, thereby reducing sequence length. However, such techniques are limited to single-context scenarios, where historical KV Cache is computed sequentially with causal-attention dependencies. In retrieval-augmented generation (RAG) scenarios, where retrieved documents as context are unknown beforehand, each document’s KV Cache is computed and stored independently (termed multiple-context KV Cache), lacking cross-attention between contexts. This renders existing methods ineffective. Although prior work partially recomputes multiple-context KV Cache to mitigate accuracy loss from missing cross-attention, it requires retaining all KV Cache throughout, failing to reduce memory overhead. This paper presents SamKV, the first exploration of attention sparsification for multiple-context KV Cache. Specifically, SamKV takes into account the complementary information of other contexts when sparsifying one context, and then locally recomputes the sparsified information. Experiments demonstrate that our method compresses sequence length to 15% without accuracy degradation compared with full-recomputation baselines, significantly boosting throughput in multi-context RAG scenarios.

Zitieren

APA

Cao, Z., Si, Q., Zhang, J., & Liu, B. (2026). Sparse Attention Across Multiple-Context KV Cache. Proceedings of the AAAI Conference on Artificial Intelligence, 40(36), 30165-30173. https://doi.org/10.1609/aaai.v40i36.40266

Vancouver

Cao Z, Si Q, Zhang J, Liu B. Sparse Attention Across Multiple-Context KV Cache. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(36):30165-30173. doi:10.1609/aaai.v40i36.40266

BibTeX
@article{Cao2026,
  author = {Cao, Ziyi and Si, Qingyi and Zhang, Jingbin and Liu, Bingquan},
  title = {Sparse Attention Across Multiple-Context KV Cache},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {36},
  pages = {30165--30173},
  doi = {10.1609/aaai.v40i36.40266}
}
Kostenlos starten →