WindowQuant: Mixed-Precision KV Cache Quantization Based on Window-Level Similarity for VLMs Inference Optimization
ACM Transactions on Architecture and Code Optimization · 30.09.2026 · 23 (3) · 1-26
DOI: 10.1145/3817053 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Architecture and Code Optimization, online erschienen am 24.07.2026. Verfasst von Wei Tao und 6 weiteren Autor:innen. Die Arbeit nennt 49 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, Special High-Quality Program of MIIT, Shandong Provincial Natural Science Foundation.
Abstract (Originalsprache)
Recently, video language models (VLMs) have been applied in various fields. However, the visual token sequence of the VLM is too long, which may cause intolerant inference latency and GPU memory usage. Existing methods propose mixed-precision quantization to the key-value (KV) cache in VLMs based on token granularity, which is time-consuming in the search process and hardware inefficient during computation. This article introduces a novel approach called WindowQuant, which employs window-adaptive mixed-precision quantization to optimize the KV cache. WindowQuant consists of two modules: window-level quantization search and window-level KV cache computation. Window-level quantization search quickly determines the optimal bit-width configuration of the KV cache windows based on the similarity scores between the corresponding visual token windows and the text prompt, maintaining the model accuracy. Furthermore, window-level KV cache computation reorders the KV cache windows before quantization, avoiding the hardware inefficiency caused by mixed-precision quantization in inference computation. Extensive experiments demonstrate that WindowQuant outperforms state-of-the-art VLM models and KV cache quantization methods on various datasets.
Förderung
- National Natural Science Foundation of China
- Special High-Quality Program of MIIT
- Shandong Provincial Natural Science Foundation
- China Postdoctoral Science Foundation
- Postdoctor Project of Hubei Province
Zitieren
APA
Tao, W., Qu, X., Wang, P., Li, G., Wan, J., Lu, K., & Wang, J. (2026). WindowQuant: Mixed-Precision KV Cache Quantization Based on Window-Level Similarity for VLMs Inference Optimization. ACM Transactions on Architecture and Code Optimization, 23(3), 1-26. https://doi.org/10.1145/3817053
Vancouver
Tao W, Qu X, Wang P, Li G, Wan J, Lu K, et al. WindowQuant: Mixed-Precision KV Cache Quantization Based on Window-Level Similarity for VLMs Inference Optimization. ACM Transactions on Architecture and Code Optimization. 2026;23(3):1-26. doi:10.1145/3817053
BibTeX
@article{Tao2026,
author = {Tao, Wei and Qu, Xiaoyang and Wang, Peiqiang and Li, Guokuan and Wan, Jiguang and Lu, Kai and Wang, Jianzong},
title = {WindowQuant: Mixed-Precision KV Cache Quantization Based on Window-Level Similarity for VLMs Inference Optimization},
journal = {ACM Transactions on Architecture and Code Optimization},
year = {2026},
volume = {23},
number = {3},
pages = {1--26},
doi = {10.1145/3817053}
}
EUVIMED