EUVIMEDEUVIMED entdecken
Crossref

UniCache: Unifying Prefix Cache Eviction for Heterogeneous LLM Serving Workloads

Bei Ouyang, Yifan Qiao, Jiarong Xing

Proceedings of the ACM on Measurement and Analysis of Computing Systems · 29.05.2026 · 10 (2) · 1-27

DOI: 10.1145/3805652 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the ACM on Measurement and Analysis of Computing Systems, online erschienen am 29.05.2026. Verfasst von Bei Ouyang und 2 weiteren Autor:innen. Die Arbeit nennt 42 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Prefix caching is a key performance optimization in Large Language Model (LLM) serving systems, enabling reuse of attention Key-Value (KV) states across requests with shared prompt prefixes. However, the size of GPU memory limits cache capacity, making the eviction policy a critical factor in overall system performance. Existing systems primarily rely on simple heuristics, such as LRU, and apply the same policy across task categories, implicitly assuming homogeneous workloads. In practice, however, modern LLMs serve heterogeneous workloads that mix multi-turn conversational traffic with diverse single-turn API requests, leading to fundamentally different prefix reuse patterns. In this work, we first design a trace-driven prefix cache simulator built on vLLM to systematically characterize prefix reuse across representative workloads. Our analysis reveals two dominant reuse patterns---session reuse and structural reuse---that vary significantly across task types. Motivated by these observations, we propose UniCache, a unified eviction policy that jointly captures both reuse patterns and dynamically balances cache allocation across tasks. When implemented in vLLM, UniCache achieves substantial improvements under heterogeneous workloads, improving prefix cache hit rates by up to 17.32% and reducing inference latency by up to 3.63× compared to existing policies.

Zitieren

APA

Ouyang, B., Qiao, Y., & Xing, J. (2026). UniCache: Unifying Prefix Cache Eviction for Heterogeneous LLM Serving Workloads. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 10(2), 1-27. https://doi.org/10.1145/3805652

Vancouver

Ouyang B, Qiao Y, Xing J. UniCache: Unifying Prefix Cache Eviction for Heterogeneous LLM Serving Workloads. Proceedings of the ACM on Measurement and Analysis of Computing Systems. 2026;10(2):1-27. doi:10.1145/3805652

BibTeX
@article{Ouyang2026,
  author = {Ouyang, Bei and Qiao, Yifan and Xing, Jiarong},
  title = {UniCache: Unifying Prefix Cache Eviction for Heterogeneous LLM Serving Workloads},
  journal = {Proceedings of the ACM on Measurement and Analysis of Computing Systems},
  year = {2026},
  volume = {10},
  number = {2},
  pages = {1--27},
  doi = {10.1145/3805652}
}
Kostenlos starten →