Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modeling
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (30) · 24982-24990
DOI: 10.1609/aaai.v40i30.39686 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Ziran Qin und 6 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref).
Abstract (Originalsprache)
Visual Autoregressive (VAR) models adopt a next-scale prediction paradigm, offering high-quality content generation with substantially fewer decoding steps. However, existing VAR models suffer from significant attention complexity and severe memory overhead due to the accumulation of key-value (KV) caches across scales. In this paper, we tackle this challenge by introducing KV cache compression into the next-scale generation paradigm. We begin with a crucial observation: attention heads in VAR models can be divided into two functionally distinct categories: Contextual Heads focus on maintaining semantic consistency, while Structural Heads are responsible for preserving spatial coherence. This structural divergence causes existing one-size-fits-all compression methods to perform poorly on VAR models. To address this, we propose HACK, a training-free Head-Aware KV cache Compression frameworK. HACK utilizes an offline classification scheme to separate head types, enabling it to apply pattern-specific compression strategies with asymmetric cache budgets for each category. By doing so, HACK effectively constrains the average KV cache length within a fixed budget B, reducing the theoretical attention complexity from O(n4) to O(Bn2). Extensive experiments on multiple VAR models across text-to-image and class-conditional tasks validate the effectiveness and generalizability of HACK. It achieves up to 70% KV cache compression without degrading output quality, resulting in memory savings and faster in- ference. For example, HACK provides a 1.75× memory reduction and a 1.57× speedup on Infinity-8B.
Zitieren
APA
Qin, Z., Lv, Y., Lin, M., Guo, H., Zhang, Z., Zou, D., & Lin, W. (2026). Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modeling. Proceedings of the AAAI Conference on Artificial Intelligence, 40(30), 24982-24990. https://doi.org/10.1609/aaai.v40i30.39686
Vancouver
Qin Z, Lv Y, Lin M, Guo H, Zhang Z, Zou D, et al. Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modeling. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(30):24982-24990. doi:10.1609/aaai.v40i30.39686
BibTeX
@article{Qin2026,
author = {Qin, Ziran and Lv, Youru and Lin, Mingbao and Guo, Hang and Zhang, Zeren and Zou, Danping and Lin, Weiyao},
title = {Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modeling},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {30},
pages = {24982--24990},
doi = {10.1609/aaai.v40i30.39686}
}
EUVIMED