EUVIMEDEUVIMED entdecken
Crossref

KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cache

Fei Li, Song Liu, Weiguo Wu, Shiqiang Nie, Jinyu Wang

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (37) · 31563-31572

DOI: 10.1609/aaai.v40i37.40422 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Fei Li und 4 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The high memory demands of the Key-Value (KV) Cache during the inference of Large Language Models (LLMs) severely restrict their deployment in resource-constrained platforms. Quantization can effectively alleviate the memory pressure caused by KV Cache. However, existing methods either rely on static one-size-fits-all precision allocation or fail to dynamically prioritize critical KV in long-context tasks, forcing memory-accuracy-throughput tradeoffs. In this work, we propose a novel mixed-precision quantization method for KV Cache named KVmix. KVmix leverages gradient-based importance analysis to evaluate how individual Key and Value projection matrices affect the model loss, enabling layer-specific bit-width allocation for mix-precision quantization. It dynamically prioritizes higher precision for important layers while aggressively quantizing less influential ones, achieving a tunable balance between accuracy and efficiency. KVmix introduces a dynamic long-context optimization strategy that adaptively keeps full-precision KV pairs for recent pivotal tokens and compresses older ones, achieving high-quality sequence generation with low memory usage. Additionally, KVmix provides efficient low-bit quantization and CUDA kernels to optimize computational overhead. On LLMs such as Llama and Mistral, KVmix achieves near-lossless inference performance with extremely low quantization configuration (Key 2.19bit Value 2.38bit), while delivering a remarkable 4.9× memory compression and a 5.3× speedup in inference throughput.

Zitieren

APA

Li, F., Liu, S., Wu, W., Nie, S., & Wang, J. (2026). KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cache. Proceedings of the AAAI Conference on Artificial Intelligence, 40(37), 31563-31572. https://doi.org/10.1609/aaai.v40i37.40422

Vancouver

Li F, Liu S, Wu W, Nie S, Wang J. KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cache. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(37):31563-31572. doi:10.1609/aaai.v40i37.40422

BibTeX
@article{Li2026,
  author = {Li, Fei and Liu, Song and Wu, Weiguo and Nie, Shiqiang and Wang, Jinyu},
  title = {KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cache},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {37},
  pages = {31563--31572},
  doi = {10.1609/aaai.v40i37.40422}
}
Kostenlos starten →