EUVIMEDEUVIMED entdecken
Crossref

CrossKV: Accelerating Large Language Model Inference via Cross-Stage Dynamic Co-Optimization for KV Cache

Nan Wu, Shenyu Wang, Huizheng Wang, Peng Wang, Xiao Liu, Zhihua Wang, Yang Hu, Hanjun Jiang

IEEE Transactions on Circuits and Systems I: Regular Papers · 2026-07 · 73 (7) · 4801-4814

DOI: 10.1109/tcsi.2026.3668423 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems I: Regular Papers, 2026-07. Verfasst von Nan Wu und 7 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 45 Literaturangaben. Gefördert von: National Key Research and Development Program, Shenzhen Science and Technology Program, Development and Reform Commission of Shenzhen Municipality.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The autoregressive nature of Large Language Models (LLMs) has enabled remarkable performance in language generation, making them a cornerstone in natural language processing. As context windows lengthen, the per-token key–value (KV) cache grows linearly with sequence length and turns the generation stage into a memory-bound operation. However, existing works focus primarily on an isolated stage of KV cache reduction, and most efforts incur significant hardware overheads. This results in a lack of cross-stage co-optimization and leaves significant reduction potential untapped. To address these challenges, we propose CrossKV, a software-hardware co-design architecture that accelerates LLM inference to fully achieve the potential of KV caching optimization. Motivated by our observation, we identify the three stages in KV caching and present a cross-stage co-optimization algorithm, including: a derivative-enhanced dynamic progressive pruning method, a DCT-driven key-vector low-rank compressing, and a dynamic clustered hybrid run-length encoding to reduce the KV caching while maintaining high accuracy. Then, an efficient architecture featuring cross-stage dynamic co-optimization with negligible hardware overhead is proposed to fully harness the algorithm of CrossKV. Our evaluations of CrossKV over 9 popular LLMs models and various long-context tasks demonstrate an average of $6.03\times $ and $4.10\times $ improvement for energy efficiency and speedup compared to existing SoTA Accelerators architecture. Compared to the Nvidia A100 GPU, CrossKV achieves an average $23.43\times $ energy efficiency and $4.64\times $ speedup, respectively.

Förderung

  • National Key Research and Development Program
  • Shenzhen Science and Technology Program
  • Development and Reform Commission of Shenzhen Municipality
  • Beijing Engineering Research Center

Zitieren

APA

Wu, N., Wang, S., Wang, H., Wang, P., Liu, X., Wang, Z., Hu, Y., & Jiang, H. (2026). CrossKV: Accelerating Large Language Model Inference via Cross-Stage Dynamic Co-Optimization for KV Cache. IEEE Transactions on Circuits and Systems I: Regular Papers, 73(7), 4801-4814. https://doi.org/10.1109/tcsi.2026.3668423

Vancouver

Wu N, Wang S, Wang H, Wang P, Liu X, Wang Z, et al. CrossKV: Accelerating Large Language Model Inference via Cross-Stage Dynamic Co-Optimization for KV Cache. IEEE Transactions on Circuits and Systems I: Regular Papers. 2026;73(7):4801-4814. doi:10.1109/tcsi.2026.3668423

BibTeX
@article{Wu2026,
  author = {Wu, Nan and Wang, Shenyu and Wang, Huizheng and Wang, Peng and Liu, Xiao and Wang, Zhihua and Hu, Yang and Jiang, Hanjun},
  title = {CrossKV: Accelerating Large Language Model Inference via Cross-Stage Dynamic Co-Optimization for KV Cache},
  journal = {IEEE Transactions on Circuits and Systems I: Regular Papers},
  year = {2026},
  volume = {73},
  number = {7},
  pages = {4801--4814},
  doi = {10.1109/tcsi.2026.3668423}
}
Kostenlos starten →