EUVIMEDEUVIMED entdecken
Crossref

Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modeling

Ziran Qin, Youru Lv, Mingbao Lin, Hang Guo, Zeren Zhang, Danping Zou, Weiyao Lin

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (30) · 24982-24990

DOI: 10.1609/aaai.v40i30.39686 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Ziran Qin und 6 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Visual Autoregressive (VAR) models adopt a next-scale prediction paradigm, offering high-quality content generation with substantially fewer decoding steps. However, existing VAR models suffer from significant attention complexity and severe memory overhead due to the accumulation of key-value (KV) caches across scales. In this paper, we tackle this challenge by introducing KV cache compression into the next-scale generation paradigm. We begin with a crucial observation: attention heads in VAR models can be divided into two functionally distinct categories: Contextual Heads focus on maintaining semantic consistency, while Structural Heads are responsible for preserving spatial coherence. This structural divergence causes existing one-size-fits-all compression methods to perform poorly on VAR models. To address this, we propose HACK, a training-free Head-Aware KV cache Compression frameworK. HACK utilizes an offline classification scheme to separate head types, enabling it to apply pattern-specific compression strategies with asymmetric cache budgets for each category. By doing so, HACK effectively constrains the average KV cache length within a fixed budget B, reducing the theoretical attention complexity from O(n4) to O(Bn2). Extensive experiments on multiple VAR models across text-to-image and class-conditional tasks validate the effectiveness and generalizability of HACK. It achieves up to 70% KV cache compression without degrading output quality, resulting in memory savings and faster in- ference. For example, HACK provides a 1.75× memory reduction and a 1.57× speedup on Infinity-8B.

Zitieren

APA

Qin, Z., Lv, Y., Lin, M., Guo, H., Zhang, Z., Zou, D., & Lin, W. (2026). Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modeling. Proceedings of the AAAI Conference on Artificial Intelligence, 40(30), 24982-24990. https://doi.org/10.1609/aaai.v40i30.39686

Vancouver

Qin Z, Lv Y, Lin M, Guo H, Zhang Z, Zou D, et al. Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modeling. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(30):24982-24990. doi:10.1609/aaai.v40i30.39686

BibTeX
@article{Qin2026,
  author = {Qin, Ziran and Lv, Youru and Lin, Mingbao and Guo, Hang and Zhang, Zeren and Zou, Danping and Lin, Weiyao},
  title = {Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modeling},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {30},
  pages = {24982--24990},
  doi = {10.1609/aaai.v40i30.39686}
}
Kostenlos starten →