KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cache
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (37) · 31563-31572
DOI: 10.1609/aaai.v40i37.40422 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Fei Li und 4 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref).
Abstract (Originalsprache)
The high memory demands of the Key-Value (KV) Cache during the inference of Large Language Models (LLMs) severely restrict their deployment in resource-constrained platforms. Quantization can effectively alleviate the memory pressure caused by KV Cache. However, existing methods either rely on static one-size-fits-all precision allocation or fail to dynamically prioritize critical KV in long-context tasks, forcing memory-accuracy-throughput tradeoffs. In this work, we propose a novel mixed-precision quantization method for KV Cache named KVmix. KVmix leverages gradient-based importance analysis to evaluate how individual Key and Value projection matrices affect the model loss, enabling layer-specific bit-width allocation for mix-precision quantization. It dynamically prioritizes higher precision for important layers while aggressively quantizing less influential ones, achieving a tunable balance between accuracy and efficiency. KVmix introduces a dynamic long-context optimization strategy that adaptively keeps full-precision KV pairs for recent pivotal tokens and compresses older ones, achieving high-quality sequence generation with low memory usage. Additionally, KVmix provides efficient low-bit quantization and CUDA kernels to optimize computational overhead. On LLMs such as Llama and Mistral, KVmix achieves near-lossless inference performance with extremely low quantization configuration (Key 2.19bit Value 2.38bit), while delivering a remarkable 4.9× memory compression and a 5.3× speedup in inference throughput.
Zitieren
APA
Li, F., Liu, S., Wu, W., Nie, S., & Wang, J. (2026). KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cache. Proceedings of the AAAI Conference on Artificial Intelligence, 40(37), 31563-31572. https://doi.org/10.1609/aaai.v40i37.40422
Vancouver
Li F, Liu S, Wu W, Nie S, Wang J. KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cache. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(37):31563-31572. doi:10.1609/aaai.v40i37.40422
BibTeX
@article{Li2026,
author = {Li, Fei and Liu, Song and Wu, Weiguo and Nie, Shiqiang and Wang, Jinyu},
title = {KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cache},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {37},
pages = {31563--31572},
doi = {10.1609/aaai.v40i37.40422}
}
EUVIMED