EUVIMEDScopri EUVIMED
Crossref

From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation

Jiahao Wang, Weiyu Xie, Mingxing Zhang, Boxin Zhang, Jianwei Dong, Yuening Zhu, Chen Lin, Jingqi Tang, Yaochen Han, Zhiyuan Ai, Xianglin Chen, Yongwei Wu and 1 more

Proceedings of the ACM on Management of Data · 2026-04-02 · 4 (1) · 1-28

DOI: 10.1145/3786655 ↗

At a glance

Journal article in Proceedings of the ACM on Management of Data, published online on 2026-04-07. Written by Jiahao Wang and 12 co-authors. Cited 2 times so far (Crossref); 72 references. Funded by: Development Program of China, National Natural Science Foundation of China, Tsinghua University Initiative Scientific Research Program, Young Elite Scientists Sponsorship Program by CAST.

AbstractDati bibliograficiCiteRelated studies

Abstract (original language)

Retrieval-Augmented Generation enhances Large Language Models by integrating external knowledge, which reduces hallucinations but increases prompt length. This increase leads to higher computational costs and longer Time to First Token. To mitigate this issue, existing solutions aim to reuse the preprocessed KVCache of each retrieved chunk to accelerate RAG. However, the lack of cross-chunk contextual information leads to a significant drop in generation quality, leaving the potential benefits of KVCache reuse largely unfulfilled. The challenge lies in how to reuse the precomputed KVCache chunk while preserving generation quality. We propose FusionRAG, a novel inference framework that optimizes both the preprocessing and reprocessing stages of RAG. In the offline preprocessing stage, we embed information from other related text chunks into each chunk, while in the online reprocessing stage, we recompute the KVCache for tokens that the model focuses on. As a result, we achieve a better trade-off between generation quality and efficiency. According to our experiments, FusionRAG significantly improves generation quality at the same recomputation ratio compared to previous state-of-the-art solutions. By recomputing fewer than 15% of the tokens, FusionRAG achieves up to 70% higher normalized-F1 scores than baselines and reduces TTFT by 2.66-9.39× compared to Full Attention.

Funding

  • Development Program of China
  • National Natural Science Foundation of China
  • Tsinghua University Initiative Scientific Research Program, Young Elite Scientists Sponsorship Program by CAST
  • Beijing Natural Science Foundation

Cite

APA

Wang, J., Xie, W., Zhang, M., Zhang, B., Dong, J., Zhu, Y., Lin, C., Tang, J., Han, Y., Ai, Z., Chen, X., Wu, Y., & Jiang, C. (2026). From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation. Proceedings of the ACM on Management of Data, 4(1), 1-28. https://doi.org/10.1145/3786655

Vancouver

Wang J, Xie W, Zhang M, Zhang B, Dong J, Zhu Y, et al. From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation. Proceedings of the ACM on Management of Data. 2026;4(1):1-28. doi:10.1145/3786655

BibTeX
@article{Wang2026,
  author = {Wang, Jiahao and Xie, Weiyu and Zhang, Mingxing and Zhang, Boxin and Dong, Jianwei and Zhu, Yuening and Lin, Chen and Tang, Jingqi and Han, Yaochen and Ai, Zhiyuan and Chen, Xianglin and Wu, Yongwei and Jiang, Congfeng},
  title = {From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation},
  journal = {Proceedings of the ACM on Management of Data},
  year = {2026},
  volume = {4},
  number = {1},
  pages = {1--28},
  doi = {10.1145/3786655}
}
Start for free →