EUVIMEDEUVIMED entdecken
Crossref

ShuffleInfer: Disaggregate LLM Inference for Mixed Downstream Workloads

Cunchen Hu, Heyang Huang, Liangliang Xu, Xusheng Chen, Chenxi Wang, Jiang Xu, Shuang Chen, Hao Feng, Sa Wang, Yungang Bao, Ninghui Sun, Yizhou Shan

ACM Transactions on Architecture and Code Optimization · 30.06.2025 · 22 (2) · 1-24

DOI: 10.1145/3732941 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Architecture and Code Optimization, 2025-07. Verfasst von Cunchen Hu und 11 weiteren Autor:innen. Die Arbeit wurde bisher 29-mal zitiert (Crossref) und nennt 62 Literaturangaben. Gefördert von: Strategic Priority Research Program of Chinese Academy of Sciences, National Natural Science Foundation of China, China Postdoctoral Science Foundation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Transformer-based large language model (LLM) inference serving is now the backbone of many cloud services. LLM inference consists of a prefill phase and a decode phase. However, existing LLM deployment practices often overlook the distinct characteristics of these phases, leading to significant interference. To mitigate interference, our insight is to carefully schedule and group inference requests based on their characteristics. We realize this idea in ShuffleInfer through three pillars. First, it partitions prompts into fixed-size chunks so that the accelerator always runs close to its computation-saturated limit. Second, it disaggregates prefill and decode instances so each can run independently. Finally, it uses a smart two-level scheduling algorithm augmented with predicted resource usage to avoid decode scheduling hotspots. Results show that ShuffleInfer improves time-to-first-token (TTFT), job completion time (JCT), and inference efficiency in terms of performance per dollar by a large margin, e.g., it uses 38% less resources all the while lowering average TTFT and average JCT by 97% and 47%, respectively.

Förderung

  • Strategic Priority Research Program of Chinese Academy of Sciences
  • National Natural Science Foundation of China
  • China Postdoctoral Science Foundation
  • Shaanxi Postdoctoral Research Foundation

Zitieren

APA

Hu, C., Huang, H., Xu, L., Chen, X., Wang, C., Xu, J., Chen, S., Feng, H., Wang, S., Bao, Y., Sun, N., & Shan, Y. (2025). ShuffleInfer: Disaggregate LLM Inference for Mixed Downstream Workloads. ACM Transactions on Architecture and Code Optimization, 22(2), 1-24. https://doi.org/10.1145/3732941

Vancouver

Hu C, Huang H, Xu L, Chen X, Wang C, Xu J, et al. ShuffleInfer: Disaggregate LLM Inference for Mixed Downstream Workloads. ACM Transactions on Architecture and Code Optimization. 2025;22(2):1-24. doi:10.1145/3732941

BibTeX
@article{Hu2025,
  author = {Hu, Cunchen and Huang, Heyang and Xu, Liangliang and Chen, Xusheng and Wang, Chenxi and Xu, Jiang and Chen, Shuang and Feng, Hao and Wang, Sa and Bao, Yungang and Sun, Ninghui and Shan, Yizhou},
  title = {ShuffleInfer: Disaggregate LLM Inference for Mixed Downstream Workloads},
  journal = {ACM Transactions on Architecture and Code Optimization},
  year = {2025},
  volume = {22},
  number = {2},
  pages = {1--24},
  doi = {10.1145/3732941}
}
Kostenlos starten →