ShuffleInfer: Disaggregate LLM Inference for Mixed Downstream Workloads
ACM Transactions on Architecture and Code Optimization · 30.06.2025 · 22 (2) · 1-24
DOI: 10.1145/3732941 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Architecture and Code Optimization, 2025-07. Verfasst von Cunchen Hu und 11 weiteren Autor:innen. Die Arbeit wurde bisher 29-mal zitiert (Crossref) und nennt 62 Literaturangaben. Gefördert von: Strategic Priority Research Program of Chinese Academy of Sciences, National Natural Science Foundation of China, China Postdoctoral Science Foundation.
Abstract (Originalsprache)
Transformer-based large language model (LLM) inference serving is now the backbone of many cloud services. LLM inference consists of a prefill phase and a decode phase. However, existing LLM deployment practices often overlook the distinct characteristics of these phases, leading to significant interference. To mitigate interference, our insight is to carefully schedule and group inference requests based on their characteristics. We realize this idea in ShuffleInfer through three pillars. First, it partitions prompts into fixed-size chunks so that the accelerator always runs close to its computation-saturated limit. Second, it disaggregates prefill and decode instances so each can run independently. Finally, it uses a smart two-level scheduling algorithm augmented with predicted resource usage to avoid decode scheduling hotspots. Results show that ShuffleInfer improves time-to-first-token (TTFT), job completion time (JCT), and inference efficiency in terms of performance per dollar by a large margin, e.g., it uses 38% less resources all the while lowering average TTFT and average JCT by 97% and 47%, respectively.
Förderung
- Strategic Priority Research Program of Chinese Academy of Sciences
- National Natural Science Foundation of China
- China Postdoctoral Science Foundation
- Shaanxi Postdoctoral Research Foundation
Zitieren
APA
Hu, C., Huang, H., Xu, L., Chen, X., Wang, C., Xu, J., Chen, S., Feng, H., Wang, S., Bao, Y., Sun, N., & Shan, Y. (2025). ShuffleInfer: Disaggregate LLM Inference for Mixed Downstream Workloads. ACM Transactions on Architecture and Code Optimization, 22(2), 1-24. https://doi.org/10.1145/3732941
Vancouver
Hu C, Huang H, Xu L, Chen X, Wang C, Xu J, et al. ShuffleInfer: Disaggregate LLM Inference for Mixed Downstream Workloads. ACM Transactions on Architecture and Code Optimization. 2025;22(2):1-24. doi:10.1145/3732941
BibTeX
@article{Hu2025,
author = {Hu, Cunchen and Huang, Heyang and Xu, Liangliang and Chen, Xusheng and Wang, Chenxi and Xu, Jiang and Chen, Shuang and Feng, Hao and Wang, Sa and Bao, Yungang and Sun, Ninghui and Shan, Yizhou},
title = {ShuffleInfer: Disaggregate LLM Inference for Mixed Downstream Workloads},
journal = {ACM Transactions on Architecture and Code Optimization},
year = {2025},
volume = {22},
number = {2},
pages = {1--24},
doi = {10.1145/3732941}
}
EUVIMED