EUVIMEDEUVIMED entdecken
Crossref

Serving Hybrid LLM Loads with SLO Guarantees Using CPU-GPU Attention Piggybacking

Zizhao Mo, Junlin Chen, Huanle Xu, ChengZhong Xu

Proceedings of the ACM on Management of Data · 18.05.2026 · 4 (3) · 1-26

DOI: 10.1145/3802107 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the ACM on Management of Data, online erschienen am 18.05.2026. Verfasst von Zizhao Mo und 3 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 64 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Nowadays, service providers often deploy multiple types of LLM services within shared clusters. While the service co-location improves resource utilization, it introduces significant interference risks for latency-sensitive (LS) services -- which have strict SLO requirements for inference latency -- and severely constrains the service capacity of best-effort (BE) services due to limited available memory. To address interference, existing systems typically rely on reserving headroom to constrain BE resource usage. However, this approach's coarse granularity compromises the SLO compliance of the latency-sensitive service and unnecessarily restricts the generation potential of the best-effort service. In this paper, we propose OmniServe, a novel LLM serving system that efficiently harnesses both CPU and GPU resources to mitigate interference and improve throughput. Central to OmniServe is the Attention Piggybacking mechanism, which effectively offloads the Attention computation of BE services to CPUs on the fly. This mechanism also facilitates asynchronous communication between CPU and GPU streams, preventing GPUs from being blocked while aggregating Attention results. Additionally, OmniServe incorporates a dynamic batching control policy to adapt to fluctuating request arrivals, facilitating Dense module computation using layer-wise batching. Experimental results show that OmniServe improves the SLO attainment rate for LS services by up to 1.48× while enhancing BE serving throughput by up to 9.85× compared to state-of-the-art systems.

Zitieren

APA

Mo, Z., Chen, J., Xu, H., & Xu, C. (2026). Serving Hybrid LLM Loads with SLO Guarantees Using CPU-GPU Attention Piggybacking. Proceedings of the ACM on Management of Data, 4(3), 1-26. https://doi.org/10.1145/3802107

Vancouver

Mo Z, Chen J, Xu H, Xu C. Serving Hybrid LLM Loads with SLO Guarantees Using CPU-GPU Attention Piggybacking. Proceedings of the ACM on Management of Data. 2026;4(3):1-26. doi:10.1145/3802107

BibTeX
@article{Mo2026,
  author = {Mo, Zizhao and Chen, Junlin and Xu, Huanle and Xu, ChengZhong},
  title = {Serving Hybrid LLM Loads with SLO Guarantees Using CPU-GPU Attention Piggybacking},
  journal = {Proceedings of the ACM on Management of Data},
  year = {2026},
  volume = {4},
  number = {3},
  pages = {1--26},
  doi = {10.1145/3802107}
}
Kostenlos starten →