EUVIMEDEUVIMED entdecken
Crossref

RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference

Yaoqi Chen, Jinkai Zhang, Baotong Lu, Qianxi Zhang, Chengruidong Zhang, Jing Liu, Jingjia Luo, Di Liu, Huiqiang Jiang, Qi Chen, Bailu Ding, Xiao Yan und 7 weitere

Proceedings of the VLDB Endowment · 01.01.2026 · 19 (5) · 1016-1031

DOI: 10.14778/3796195.3796212 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the VLDB Endowment, online erschienen am 05.05.2026. Verfasst von Yaoqi Chen und 18 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 126 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recent large language models (LLMs) are rapidly extending their context windows, yet inference throughput lags due to increasing GPU memory and bandwidth demands. This is because the key-value (KV) cache, an intermediate structure storing token representations, grows linearly with context length and requires an iterative linear scan for attention computation. A promising direction to accelerate long-context inference is to exploit attention's inherent sparsity by offloading the KV cache to CPU memory and retrieving only a small subset of tokens important to the current generation step. However, prior sparse attention approaches struggle to balance accuracy and retrieval cost due to varying sparsity patterns and inefficient GPU-CPU memory management. We present RetroInfer, a vector storage engine that realizes a sparsity-based KV cache for long-context inference. RetroInfer introduces an Attention-aWare VEctor index ( wave index ), which fundamentally improves the tradeoff between attention accuracy and retrieval cost through tripartite attention approximation, accuracy-bound attention estimation, and segmented clustering. We also design the wave buffer , a GPU-CPU buffer manager that assigns computation and manages data across heterogeneous hardware. We evaluate RetroInfer across a range of models and workloads, demonstrating up to 4.4× decoding throughput over full attention at 120K context and up to 12.2× over sparse attention baselines at 1 million tokens—all while preserving full-attention-level accuracy.

Zitieren

APA

Chen, Y., Zhang, J., Lu, B., Zhang, Q., Zhang, C., Liu, J., Luo, J., Liu, D., Jiang, H., Chen, Q., Ding, B., Yan, X., Jiang, J., Chen, C., Zhang, M., Li, C., Yang, Y., Yang, F., & Yang, M. (2026). RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference. Proceedings of the VLDB Endowment, 19(5), 1016-1031. https://doi.org/10.14778/3796195.3796212

Vancouver

Chen Y, Zhang J, Lu B, Zhang Q, Zhang C, Liu J, et al. RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference. Proceedings of the VLDB Endowment. 2026;19(5):1016-1031. doi:10.14778/3796195.3796212

BibTeX
@article{Chen2026,
  author = {Chen, Yaoqi and Zhang, Jinkai and Lu, Baotong and Zhang, Qianxi and Zhang, Chengruidong and Liu, Jing and Luo, Jingjia and Liu, Di and Jiang, Huiqiang and Chen, Qi and Ding, Bailu and Yan, Xiao and Jiang, Jiawei and Chen, Chen and Zhang, Mingxing and Li, Cheng and Yang, Yuqing and Yang, Fan and Yang, Mao},
  title = {RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference},
  journal = {Proceedings of the VLDB Endowment},
  year = {2026},
  volume = {19},
  number = {5},
  pages = {1016--1031},
  doi = {10.14778/3796195.3796212}
}
Kostenlos starten →