EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling

Shashwat Jaiswal, Kunal Jain, Yogesh Simmhan, Anjaly Parayil, Ankur Mallick, Rujia Wang, Renee St. Amant, Chetan Bansal, Victor Rühle, Anoop Kulkarni, Steve Kofsky, Saravan Rajmohan

ACM SIGMETRICS Performance Evaluation Review · 20.07.2026 · 54 (1) · 232-234

DOI: 10.1145/3820171.3806875 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM SIGMETRICS Performance Evaluation Review, online erschienen am 27.07.2026. Verfasst von Shashwat Jaiswal und 11 weiteren Autor:innen. Die Arbeit nennt 10 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Global cloud service providers handle inference workloads for LLMs that span latency-sensitive (e.g., chatbots) and insensitive (e.g., report writing) tasks, with diverse and often conflicting Service Level Agreement (SLA) needs. Serving such mixed workloads is challenging due to the multiple models, diverse GPU hardware, and globally distributed data centers. Solutions that silo fast and slow tasks on separate GPU pools cause under-utilization of expensive accelerators. Here, we characterize the large-scale LLM inference workloads (> 10 M requests per day) at Microsoft Office 365, and highlight key observations in one of the first Internet-scale public studies. These insights drive SageServe , an LLM serving framework that dynamically adapts to workload demands. It combines short-term request routing to data centers with long-term scaling of GPU VMs and model placement with higher lead times, and co-optimizes the routing and resource allocation problem using a traffic forecast model and an ILP solution. We evaluate SageServe on traces comprising 10M production requests across three regions and four open-source models. SageServe achieves up to 25% savings in GPU-hours compared to the current baseline deployment and reduces GPU-hour wastage due to inefficient auto-scaling by 80%, potentially resulting in monthly cost savings of up to $2.5 million, while maintaining tail latency and meeting SLAs.

Zitieren

APA

Jaiswal, S., Jain, K., Simmhan, Y., Parayil, A., Mallick, A., Wang, R., Amant, R. S., Bansal, C., Rühle, V., Kulkarni, A., Kofsky, S., & Rajmohan, S. (2026). SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling. ACM SIGMETRICS Performance Evaluation Review, 54(1), 232-234. https://doi.org/10.1145/3820171.3806875

Vancouver

Jaiswal S, Jain K, Simmhan Y, Parayil A, Mallick A, Wang R, et al. SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling. ACM SIGMETRICS Performance Evaluation Review. 2026;54(1):232-234. doi:10.1145/3820171.3806875

BibTeX
@article{Jaiswal2026,
  author = {Jaiswal, Shashwat and Jain, Kunal and Simmhan, Yogesh and Parayil, Anjaly and Mallick, Ankur and Wang, Rujia and Amant, Renee St. and Bansal, Chetan and Rühle, Victor and Kulkarni, Anoop and Kofsky, Steve and Rajmohan, Saravan},
  title = {SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling},
  journal = {ACM SIGMETRICS Performance Evaluation Review},
  year = {2026},
  volume = {54},
  number = {1},
  pages = {232--234},
  doi = {10.1145/3820171.3806875}
}
Kostenlos starten →