EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving

Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Heyi Tang, Feng Ren, Teng Ma, Shangming Cai, Yineng Zhang, Mingxing Zhang, Yongwei Wu, Weimin Zheng und 1 weitere

ACM Transactions on Storage · 30.12.2026 · 22 (4) · 1-38

DOI: 10.1145/3773772 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Storage, online erschienen am 17.08.2026. Verfasst von Ruoyu Qin und 12 weiteren Autor:innen. Die Arbeit wurde bisher 32-mal zitiert (Crossref) und nennt 52 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Key Research & Development Program of China, Beijing Natural Science Foundation, Tsinghua University Initiative Scientific Research Program.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Mooncake is the serving platform for Kimi, an LLM chatbot service developed by Moonshot AI. This platform features a KVCache-centric disaggregated architecture that not only separates prefill and decoding clusters but also efficiently utilizes the underexploited CPU, DRAM, SSD and NIC resources of the GPU cluster to establish a disaggregated KVCache. At the core of Mooncake is its KVCache-centric global cache and a scheduler designed to maximize throughput while adhering to stringent latency-related Service Level Objectives (SLOs). Our experiments demonstrate that Mooncake excels in scenarios involving long-context inputs. In tests using real traces, Mooncake increases the effective request capacity by 59%∼498% when compared to baseline methods, all while complying with SLOs. Currently, Mooncake is operational across thousands of nodes, processing over 100 billion tokens daily. In practical deployments, Mooncake ’s innovative architecture enables Kimi to handle 115% and 107% more requests on NVIDIA A800 and H800 clusters, respectively, compared to previous systems.

Förderung

  • National Key Research & Development Program of China
  • Beijing Natural Science Foundation
  • Tsinghua University Initiative Scientific Research Program
  • Approaching.AI Co., Ltd.

Zitieren

APA

Qin, R., Li, Z., He, W., Cui, J., Tang, H., Ren, F., Ma, T., Cai, S., Zhang, Y., Zhang, M., Wu, Y., Zheng, W., & Xu, X. (2026). Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. ACM Transactions on Storage, 22(4), 1-38. https://doi.org/10.1145/3773772

Vancouver

Qin R, Li Z, He W, Cui J, Tang H, Ren F, et al. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving. ACM Transactions on Storage. 2026;22(4):1-38. doi:10.1145/3773772

BibTeX
@article{Qin2026,
  author = {Qin, Ruoyu and Li, Zheming and He, Weiran and Cui, Jialei and Tang, Heyi and Ren, Feng and Ma, Teng and Cai, Shangming and Zhang, Yineng and Zhang, Mingxing and Wu, Yongwei and Zheng, Weimin and Xu, Xinran},
  title = {Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving},
  journal = {ACM Transactions on Storage},
  year = {2026},
  volume = {22},
  number = {4},
  pages = {1--38},
  doi = {10.1145/3773772}
}
Kostenlos starten →