EUVIMEDEUVIMED entdecken
Crossref

KVD rive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference

Jian Lin, Jiazhi Mi, Zicong Hong, Haodong Wang, Qianli Liu, Haoyue Zhang, Peng Li, Song Guo

Proceedings of the ACM on Management of Data · 18.05.2026 · 4 (3) · 1-25

DOI: 10.1145/3802077 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the ACM on Management of Data, online erschienen am 18.05.2026. Verfasst von Jian Lin und 7 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 41 Literaturangaben. Gefördert von: Hong Kong RGC General Research Fund, Natural Science Foundation of China (NSFC) Key Program, Collaborative Research Fund.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Supporting long-context LLMs is challenging due to the substantial memory demands of the key–value (KV) cache. Existing offloading systems store the full cache in host memory and selectively fetch critical entries during decoding, but this strategy quickly hits a ceiling: sparsity cannot be pushed further without degrading accuracy. As a result, when context length and batch size grow, the volume of KV transfers rises sharply and becomes the dominant source of decoding latency. We present KVD rive, a holistic multi-tier KV cache management system spanning GPU memory, host DRAM, and SSD. Unlike prior work that pursues greater sparsity through algorithmic refinements, KVD rive tackles the problem from a systems perspective—jointly orchestrating cache placement, pipeline scheduling, and cross-tier coordination to sustain high-throughput inference under tight GPU budgets. KVD rive advances three fundamental capabilities: it adapts cache management to attention behavior to maximize reuse and minimize redundant data movement; it restructures the decoding pipeline to overlap I/O- and CPU/GPU compute-bound stages, eliminating stalls across heterogeneous resources; and it harmonizes data movement across memory tiers to unlock scalable long-context inference far beyond GPU and DRAM limits. We have implemented a fully functional prototype of KVD rive and evaluated it on long-context benchmarks with popular LLMs. The system achieves up to 1.74× higher throughput compared to state-of-the-art works while preserving accuracy.

Förderung

  • Hong Kong RGC General Research Fund
  • Natural Science Foundation of China (NSFC) Key Program
  • Collaborative Research Fund
  • NSFC\/RGC Collaborative Research Scheme
  • Research Impact Fund
  • Areas of Excellence Scheme
  • InnoHK
  • National Natural Science Foundation of China

Zitieren

APA

Lin, J., Mi, J., Hong, Z., Wang, H., Liu, Q., Zhang, H., Li, P., & Guo, S. (2026). KVD rive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference. Proceedings of the ACM on Management of Data, 4(3), 1-25. https://doi.org/10.1145/3802077

Vancouver

Lin J, Mi J, Hong Z, Wang H, Liu Q, Zhang H, et al. KVD rive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference. Proceedings of the ACM on Management of Data. 2026;4(3):1-25. doi:10.1145/3802077

BibTeX
@article{Lin2026,
  author = {Lin, Jian and Mi, Jiazhi and Hong, Zicong and Wang, Haodong and Liu, Qianli and Zhang, Haoyue and Li, Peng and Guo, Song},
  title = {KVD rive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference},
  journal = {Proceedings of the ACM on Management of Data},
  year = {2026},
  volume = {4},
  number = {3},
  pages = {1--25},
  doi = {10.1145/3802077}
}
Kostenlos starten →