Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management
Proceedings of the ACM on Management of Data · 02.04.2026 · 4 (1) · 1-29
DOI: 10.1145/3786627 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the ACM on Management of Data, online erschienen am 07.04.2026. Verfasst von Xinjun Yang und 11 weiteren Autor:innen. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 76 Literaturangaben.
Abstract (Originalsprache)
The rapid increase in LLM model sizes and the growing demand for long-context inference have made memory a critical bottleneck in GPU-accelerated LLM serving. Although high-bandwidth memory (HBM) on GPUs offers fast access, its limited capacity necessitates reliance on host memory (CPU DRAM) to support large KVCache. However, the maximum DRAM capacity is constrained by the limited number of memory channels per CPU socket. To overcome this limitation, current systems often adopt RDMA-based disaggregated memory pools, which introduce significant challenges including high access latency, complex communication protocols, and synchronization overhead. Fortunately, the emerging CXL technology introduces new opportunities in KVCache design. In this paper, we propose Beluga , a novel memory architecture that enables GPUs and CPUs to access a shared, large-scale memory pool through CXL switches. By supporting native load/store access semantics over the CXL fabric, our design delivers near-local memory latency, while reducing programming complexity and minimizing synchronization overhead. We conduct a systematic characterization of CXL-based memory pool and propose a set of design guidelines. Based on Beluga , we design and implement Beluga -KVCache, a system tailored for managing the large-scale KVCache for LLM inference. Beluga -KVCache achieves an 89.6% reduction in Time-To-First-Token (TTFT) and 7.35x throughput improvement in vLLM compared to RDMA-based solutions. To the best of our knowledge, Beluga is the first system that enables GPUs to directly access large-scale memory pools through CXL switches (Marvell XConn XC50256), marking a significant step toward low-latency, shared access to vast memory resources by GPUs.
Zitieren
APA
Yang, X., Hu, Q., Li, J., Li, F., Zhu, Y., Zhou, Y., Lin, Q., Dai, J., Kong, Y., Zhang, J., Xu, G., & Liu, Q. (2026). Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management. Proceedings of the ACM on Management of Data, 4(1), 1-29. https://doi.org/10.1145/3786627
Vancouver
Yang X, Hu Q, Li J, Li F, Zhu Y, Zhou Y, et al. Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management. Proceedings of the ACM on Management of Data. 2026;4(1):1-29. doi:10.1145/3786627
BibTeX
@article{Yang2026,
author = {Yang, Xinjun and Hu, Qingda and Li, Junru and Li, Feifei and Zhu, Yicong and Zhou, Yuqi and Lin, Qiuru and Dai, Jian and Kong, Yang and Zhang, Jiayu and Xu, Guoqiang and Liu, Qiang},
title = {Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management},
journal = {Proceedings of the ACM on Management of Data},
year = {2026},
volume = {4},
number = {1},
pages = {1--29},
doi = {10.1145/3786627}
}
EUVIMED