UniCache: Unifying Prefix Cache Eviction for Heterogeneous LLM Serving Workloads
ACM SIGMETRICS Performance Evaluation Review · 20.07.2026 · 54 (1) · 31-33
DOI: 10.1145/3820171.3806861 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM SIGMETRICS Performance Evaluation Review, online erschienen am 27.07.2026. Verfasst von Bei Ouyang und 2 weiteren Autor:innen. Die Arbeit nennt 9 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Prefix caching is a key performance optimization in Large Language Model (LLM) serving systems, enabling reuse of attention Key-Value (KV) states across requests with shared prompt prefixes. However, the size of GPU memory limits cache capacity, making the eviction policy a critical factor in overall system performance. Existing systems primarily rely on simple heuristics, such as LRU, and apply the same policy across task categories, implicitly assuming homogeneous workloads. In practice, however, modern LLMs serve heterogeneous workloads that mix multi-turn conversational traffic with diverse single-turn API requests, leading to fundamentally different prefix reuse patterns. In this work, we first design a trace-driven prefix cache simulator built on vLLM to systematically characterize prefix reuse across representative workloads. Our analysis reveals two dominant reuse patterns---session reuse and structural reuse---that vary significantly across task types. Motivated by these observations, we propose \sys, a unified eviction policy that jointly captures both reuse patterns and dynamically balances cache allocation across tasks. When implemented in vLLM, UniCache achieves substantial improvements under heterogeneous workloads, improving prefix cache hit rates by up to 17.32% and reducing inference latency by up to 3.63× compared to existing policies.
Zitieren
APA
Ouyang, B., Qiao, Y., & Xing, J. (2026). UniCache: Unifying Prefix Cache Eviction for Heterogeneous LLM Serving Workloads. ACM SIGMETRICS Performance Evaluation Review, 54(1), 31-33. https://doi.org/10.1145/3820171.3806861
Vancouver
Ouyang B, Qiao Y, Xing J. UniCache: Unifying Prefix Cache Eviction for Heterogeneous LLM Serving Workloads. ACM SIGMETRICS Performance Evaluation Review. 2026;54(1):31-33. doi:10.1145/3820171.3806861
BibTeX
@article{Ouyang2026,
author = {Ouyang, Bei and Qiao, Yifan and Xing, Jiarong},
title = {UniCache: Unifying Prefix Cache Eviction for Heterogeneous LLM Serving Workloads},
journal = {ACM SIGMETRICS Performance Evaluation Review},
year = {2026},
volume = {54},
number = {1},
pages = {31--33},
doi = {10.1145/3820171.3806861}
}
EUVIMED