FlexKV: Resource-Efficient Remote KV Reuse for Low-Latency LLM Inference
IEEE Transactions on Parallel and Distributed Systems · 2026-12 · 37 (12) · 2867-2880
DOI: 10.1109/tpds.2026.3746050 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Parallel and Distributed Systems, 2026-12. Verfasst von Guotao Yang und 7 weiteren Autor:innen. Die Arbeit nennt 47 Literaturangaben. Gefördert von: National Natural Science Foundation of China.
Abstract (Originalsprache)
Large language models (LLMs) have demonstrated exceptional performance across a variety of online services, which require real-time responses with low time-to-first-token (TTFT). To reduce TTFT, existing works commonly reuse the KV cache for requests sharing identical prefixes and leverage remote KV caches to increase the reuse rate. Existing remote KV-cache reuse policies, which we call resource-insensitive reuse policies, assume the network is stable and ignore real-time resource fluctuations. However, such policies cannot maintain low TTFT, as they neither make dynamic selections among different reuse representations nor leverage hybrid reuse to increase resource utilization, leading to significant waste of compute and transfer resources. We propose that an efficient LLM inference system should adopt a resource-efficient KV-cache reuse policy to reduce TTFT. To do so, we design an LLM inference system, FlexKV, which decreases TTFT while achieving high resource utilization by integrating two novel components: an adaptive Hybrid Adapter that chooses which reuse approaches to utilize, and a layer-wise Pipeline Scheduler that coordinates multiple compute and transfer operations. Evaluation across 14 unique model–workload combinations spanning 6.7B to 65B models, including a multi-node LLaMA-65B-GPTQ deployment, shows that FlexKV achieves up to a 4.90× TTFT speedup over evaluated deployable baselines. Compared with the analytical Optimal reference, FlexKV has a median TTFT gap of 8.10%.
Förderung
- National Natural Science Foundation of China
Zitieren
APA
Yang, G., Wang, X., Hu, Y., Yin, L., Wang, Z., Guo, R., Chen, S., & Li, K. (2026). FlexKV: Resource-Efficient Remote KV Reuse for Low-Latency LLM Inference. IEEE Transactions on Parallel and Distributed Systems, 37(12), 2867-2880. https://doi.org/10.1109/tpds.2026.3746050
Vancouver
Yang G, Wang X, Hu Y, Yin L, Wang Z, Guo R, et al. FlexKV: Resource-Efficient Remote KV Reuse for Low-Latency LLM Inference. IEEE Transactions on Parallel and Distributed Systems. 2026;37(12):2867-2880. doi:10.1109/tpds.2026.3746050
BibTeX
@article{Yang2026,
author = {Yang, Guotao and Wang, Xinpei and Hu, Yitao and Yin, Longfei and Wang, Zhengchao and Guo, Rui and Chen, Sheng and Li, Keqiu},
title = {FlexKV: Resource-Efficient Remote KV Reuse for Low-Latency LLM Inference},
journal = {IEEE Transactions on Parallel and Distributed Systems},
year = {2026},
volume = {37},
number = {12},
pages = {2867--2880},
doi = {10.1109/tpds.2026.3746050}
}
EUVIMED