Elastic Memory Remapping for Multi-tenant LLM Serving
ACM SIGOPS Operating Systems Review · 13.07.2026 · 60 (1) · 41-49
DOI: 10.1145/3830422.3830427 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM SIGOPS Operating Systems Review, online erschienen am 13.07.2026. Verfasst von Ruihao Li und 6 weiteren Autor:innen. Die Arbeit nennt 28 Literaturangaben.
Abstract (Originalsprache)
KV cache accelerates LLM inference by avoiding redundant computation, but its rapidly growing memory footprint makes GPU memory a primary bottleneck in modern serving systems. Recent approaches extend GPU memory using CPU memory through KV-cache swapping. However, because KV cache is continuously updated during decoding, swapping introduces substantial synchronization and bidirectional transfer overheads. We present Oneiros, a dynamic remapping engine for multi-tenant LLM serving. Oneiros is based on a simple observation: unlike KV cache, model parameters remain immutable during inference. Instead of swapping KV cache itself, Oneiros dynamically repurposes GPU memory allocated for model parameters as KV cache capacity, enabling nonblocking, unidirectional parameter transfer. This approach is particularly effective in multi-tenant environments, where memory allocated to inactive models can be reclaimed dynamically for active workloads. We implement Oneiros in vLLM and evaluate it on modern GH200 systems. Compared to vLLM, Oneiros reduces tail latency by up to 99.3% and improves throughput by up to 86.7%. Compared to KV-cache swapping approaches, Oneiros achieves substantially higher throughput by avoiding synchronization overheads during runtime memory extension. Source code of Oneiros is available at https://github.com/UT-SysML/Oneiros/1.
Zitieren
APA
Li, R., Pal, S., Pullu, V. N., Sinha, P., Ryoo, J., John, L. K., & Yadwadkar, N. J. (2026). Elastic Memory Remapping for Multi-tenant LLM Serving. ACM SIGOPS Operating Systems Review, 60(1), 41-49. https://doi.org/10.1145/3830422.3830427
Vancouver
Li R, Pal S, Pullu VN, Sinha P, Ryoo J, John LK, et al. Elastic Memory Remapping for Multi-tenant LLM Serving. ACM SIGOPS Operating Systems Review. 2026;60(1):41-49. doi:10.1145/3830422.3830427
BibTeX
@article{Li2026,
author = {Li, Ruihao and Pal, Shagnik and Pullu, Vineeth Narayan and Sinha, Prasoon and Ryoo, Jeeho and John, Lizy K. and Yadwadkar, Neeraja J.},
title = {Elastic Memory Remapping for Multi-tenant LLM Serving},
journal = {ACM SIGOPS Operating Systems Review},
year = {2026},
volume = {60},
number = {1},
pages = {41--49},
doi = {10.1145/3830422.3830427}
}
EUVIMED