OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration
Proceedings of the VLDB Endowment · 01.01.2026 · 19 (5) · 1046-1059
DOI: 10.14778/3796195.3796214 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the VLDB Endowment, online erschienen am 05.05.2026. Verfasst von Xinyue Ma und 6 weiteren Autor:innen. Die Arbeit nennt 56 Literaturangaben.
Abstract (Originalsprache)
Serving long-context LLMs is challenging because request lengths and batch composition vary during token generation, causing the memory footprint to fluctuate significantly at runtime. Offloading KV caches to host memory limits effective GPU memory usage, but existing static and predetermined offloading strategies cannot adapt to the rapidly shifting memory demands of long-context serving. This often leads to excessive CPU-to-GPU KV transfers that translate into latency spikes and frequent SLO violations. To address these challenges, we introduce OrbitFlow, a finegrained and adaptive KV cache management system that meets latency SLOs in long-context LLM serving. OrbitFlow employs a lightweight ILP solver to decide which layers' KV caches to retain on the GPU for each request, within memory capacity constraints. It continuously refines KV placements based on runtime feedback when the active plan becomes suboptimal during token generation. Under heavy load, OrbitFlow invokes a fallback mechanism to temporarily defer in-flight requests with large memory footprints, preserving overall SLO attainment. Our experiments demonstrate that OrbitFlow improves SLO attainment for TPOT and TBT by 62% and 66%, respectively, while reducing the 95th percentile (i.e., P95) latency by 38% and achieving up to 3.3× higher throughput compared to existing offloading methods.
Zitieren
APA
Ma, X., Hong, H., Um, T., Lee, J., Choy, S., Lee, W., & Jeon, M. (2026). OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration. Proceedings of the VLDB Endowment, 19(5), 1046-1059. https://doi.org/10.14778/3796195.3796214
Vancouver
Ma X, Hong H, Um T, Lee J, Choy S, Lee W, et al. OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration. Proceedings of the VLDB Endowment. 2026;19(5):1046-1059. doi:10.14778/3796195.3796214
BibTeX
@article{Ma2026,
author = {Ma, Xinyue and Hong, Heelim and Um, Taegeon and Lee, Jongseop and Choy, Seoyeong and Lee, Woo-Yeon and Jeon, Myeongjae},
title = {OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration},
journal = {Proceedings of the VLDB Endowment},
year = {2026},
volume = {19},
number = {5},
pages = {1046--1059},
doi = {10.14778/3796195.3796214}
}
EUVIMED