AdaptiveKV: Accelerating KV Cache Offloading with a Bandwidth-Adaptive Memory Allocation Mechanism
ACM Transactions on Architecture and Code Optimization · 30.09.2026 · 23 (3) · 1-25
DOI: 10.1145/3821219 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Architecture and Code Optimization, online erschienen am 07.08.2026. Verfasst von Yibo Tang und 10 weiteren Autor:innen. Die Arbeit nennt 61 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, National Key Research and Development Program of China.
Abstract (Originalsprache)
The explosive growth of key-value (KV) cache size in large language model (LLM) inference poses a key challenge to the limited HBM of GPU. Offloading KV cache to host memory has become a prevalent mitigation method. However, the limited host DDR bandwidth, especially in multi-GPU inference scenarios, often leads to offloading bottlenecks, thereby restricting inference speed. Compute express link (CXL) offers a promising alternative to expand host memory capacity and bandwidth on demand. In this article, we present a bandwidth-oriented memory allocation mechanism, named AdaptiveKV, which is self-adaptive to CXL-enabled memory pools and KV cache offloading scales for LLM inference acceleration. Our systematic profiling of CXL-HBM memory bandwidth under GPU workloads reveals that conventional memory strategies neglect dynamic memory bandwidth fluctuations and various CXL memory characteristics, leading to suboptimal memory utilization. Motivated by these insights, AdaptiveKV implements three core designs: (1) a GPU memory conch model to guide memory allocation strategies, (2) a runtime predictor to predict optimal memory allocation ratios, and (3) a dynamic interleaving strategy to allocate memory pages across available NUMA nodes. Experimental results suggest that AdaptiveKV achieves a maximum speedup of 1.90× in LLM inference throughput compared with the state-of-the-art strategies. To further explore AdaptiveKV’s applicability boundary, we also present an FPGA-based CXL memory emulator with configurable performance, revealing that a CXL-to-DDR bandwidth ratio exceeding 8% yields at least a 5% speedup in LLM inference.
Förderung
- National Natural Science Foundation of China
- National Key Research and Development Program of China
Zitieren
APA
Tang, Y., Wu, L., Ou, Y., Gao, S., Wang, Y., Wang, Z., Qi, X., Xu, J., Lv, F., Xiao, L., & Lai, M. (2026). AdaptiveKV: Accelerating KV Cache Offloading with a Bandwidth-Adaptive Memory Allocation Mechanism. ACM Transactions on Architecture and Code Optimization, 23(3), 1-25. https://doi.org/10.1145/3821219
Vancouver
Tang Y, Wu L, Ou Y, Gao S, Wang Y, Wang Z, et al. AdaptiveKV: Accelerating KV Cache Offloading with a Bandwidth-Adaptive Memory Allocation Mechanism. ACM Transactions on Architecture and Code Optimization. 2026;23(3):1-25. doi:10.1145/3821219
BibTeX
@article{Tang2026,
author = {Tang, Yibo and Wu, Lizhou and Ou, Yang and Gao, Sunfeng and Wang, Yanjing and Wang, Zicong and Qi, Xingyun and Xu, Jiaqing and Lv, Fangxu and Xiao, Liquan and Lai, Mingche},
title = {AdaptiveKV: Accelerating KV Cache Offloading with a Bandwidth-Adaptive Memory Allocation Mechanism},
journal = {ACM Transactions on Architecture and Code Optimization},
year = {2026},
volume = {23},
number = {3},
pages = {1--25},
doi = {10.1145/3821219}
}
EUVIMED