EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

WindowQuant: Mixed-Precision KV Cache Quantization Based on Window-Level Similarity for VLMs Inference Optimization

Wei Tao, Xiaoyang Qu, Peiqiang Wang, Guokuan Li, Jiguang Wan, Kai Lu, Jianzong Wang

ACM Transactions on Architecture and Code Optimization · 30.09.2026 · 23 (3) · 1-26

DOI: 10.1145/3817053 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Architecture and Code Optimization, online erschienen am 24.07.2026. Verfasst von Wei Tao und 6 weiteren Autor:innen. Die Arbeit nennt 49 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, Special High-Quality Program of MIIT, Shandong Provincial Natural Science Foundation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recently, video language models (VLMs) have been applied in various fields. However, the visual token sequence of the VLM is too long, which may cause intolerant inference latency and GPU memory usage. Existing methods propose mixed-precision quantization to the key-value (KV) cache in VLMs based on token granularity, which is time-consuming in the search process and hardware inefficient during computation. This article introduces a novel approach called WindowQuant, which employs window-adaptive mixed-precision quantization to optimize the KV cache. WindowQuant consists of two modules: window-level quantization search and window-level KV cache computation. Window-level quantization search quickly determines the optimal bit-width configuration of the KV cache windows based on the similarity scores between the corresponding visual token windows and the text prompt, maintaining the model accuracy. Furthermore, window-level KV cache computation reorders the KV cache windows before quantization, avoiding the hardware inefficiency caused by mixed-precision quantization in inference computation. Extensive experiments demonstrate that WindowQuant outperforms state-of-the-art VLM models and KV cache quantization methods on various datasets.

Förderung

  • National Natural Science Foundation of China
  • Special High-Quality Program of MIIT
  • Shandong Provincial Natural Science Foundation
  • China Postdoctoral Science Foundation
  • Postdoctor Project of Hubei Province

Zitieren

APA

Tao, W., Qu, X., Wang, P., Li, G., Wan, J., Lu, K., & Wang, J. (2026). WindowQuant: Mixed-Precision KV Cache Quantization Based on Window-Level Similarity for VLMs Inference Optimization. ACM Transactions on Architecture and Code Optimization, 23(3), 1-26. https://doi.org/10.1145/3817053

Vancouver

Tao W, Qu X, Wang P, Li G, Wan J, Lu K, et al. WindowQuant: Mixed-Precision KV Cache Quantization Based on Window-Level Similarity for VLMs Inference Optimization. ACM Transactions on Architecture and Code Optimization. 2026;23(3):1-26. doi:10.1145/3817053

BibTeX
@article{Tao2026,
  author = {Tao, Wei and Qu, Xiaoyang and Wang, Peiqiang and Li, Guokuan and Wan, Jiguang and Lu, Kai and Wang, Jianzong},
  title = {WindowQuant: Mixed-Precision KV Cache Quantization Based on Window-Level Similarity for VLMs Inference Optimization},
  journal = {ACM Transactions on Architecture and Code Optimization},
  year = {2026},
  volume = {23},
  number = {3},
  pages = {1--26},
  doi = {10.1145/3817053}
}
Kostenlos starten →