SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling
ACM SIGMETRICS Performance Evaluation Review · 20.07.2026 · 54 (1) · 232-234
DOI: 10.1145/3820171.3806875 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM SIGMETRICS Performance Evaluation Review, online erschienen am 27.07.2026. Verfasst von Shashwat Jaiswal und 11 weiteren Autor:innen. Die Arbeit nennt 10 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).
Abstract (Originalsprache)
Global cloud service providers handle inference workloads for LLMs that span latency-sensitive (e.g., chatbots) and insensitive (e.g., report writing) tasks, with diverse and often conflicting Service Level Agreement (SLA) needs. Serving such mixed workloads is challenging due to the multiple models, diverse GPU hardware, and globally distributed data centers. Solutions that silo fast and slow tasks on separate GPU pools cause under-utilization of expensive accelerators. Here, we characterize the large-scale LLM inference workloads (> 10 M requests per day) at Microsoft Office 365, and highlight key observations in one of the first Internet-scale public studies. These insights drive SageServe , an LLM serving framework that dynamically adapts to workload demands. It combines short-term request routing to data centers with long-term scaling of GPU VMs and model placement with higher lead times, and co-optimizes the routing and resource allocation problem using a traffic forecast model and an ILP solution. We evaluate SageServe on traces comprising 10M production requests across three regions and four open-source models. SageServe achieves up to 25% savings in GPU-hours compared to the current baseline deployment and reduces GPU-hour wastage due to inefficient auto-scaling by 80%, potentially resulting in monthly cost savings of up to $2.5 million, while maintaining tail latency and meeting SLAs.
Zitieren
APA
Jaiswal, S., Jain, K., Simmhan, Y., Parayil, A., Mallick, A., Wang, R., Amant, R. S., Bansal, C., Rühle, V., Kulkarni, A., Kofsky, S., & Rajmohan, S. (2026). SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling. ACM SIGMETRICS Performance Evaluation Review, 54(1), 232-234. https://doi.org/10.1145/3820171.3806875
Vancouver
Jaiswal S, Jain K, Simmhan Y, Parayil A, Mallick A, Wang R, et al. SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling. ACM SIGMETRICS Performance Evaluation Review. 2026;54(1):232-234. doi:10.1145/3820171.3806875
BibTeX
@article{Jaiswal2026,
author = {Jaiswal, Shashwat and Jain, Kunal and Simmhan, Yogesh and Parayil, Anjaly and Mallick, Ankur and Wang, Rujia and Amant, Renee St. and Bansal, Chetan and Rühle, Victor and Kulkarni, Anoop and Kofsky, Steve and Rajmohan, Saravan},
title = {SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling},
journal = {ACM SIGMETRICS Performance Evaluation Review},
year = {2026},
volume = {54},
number = {1},
pages = {232--234},
doi = {10.1145/3820171.3806875}
}
EUVIMED