BlitzScale: Fast and Live Large Model Autoscaling with O(1) Host Caching
ACM Transactions on Computer Systems · 08.05.2026
DOI: 10.1145/3807606 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Computer Systems, online erschienen am 08.05.2026. Verfasst von Dingyan Zhang und 6 weiteren Autor:innen. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 90 Literaturangaben.
Abstract (Originalsprache)
Model autoscaling is the key mechanism for serverless model-as-a-service, but faces a fundamental trade-off between scaling speed and storage/memory usage for caching parameters, and cannot meet frequent multi-host scaling demands. The root cause is a slow, blocking data plane: scaled instances stop while parameters load. In this paper, we first show that the data plane—loading model checkpoints to accelerators—can be made fast with no or O (1) caching, by loading parameters through the inter-GPU compute network: (1) its speed is comparable to host cache yet underutilized, and (2) scaling multiple instances needs no or O (1) caching via network-optimized multicast. Second, autoscaling can be made live by shifting the scaling abstraction from coarse-grained instance-level to fine-grained layer-level, allowing us to offload layer computation from overloaded instances to scaled ones before parameters fully load. Under real-world workloads, BlitzScale achieves up to 94 % lower tail latency than the state-of-the-art autoscaling system (ServerlessLLM), and cuts serving GPU time by 49 % versus non-autoscaling systems like DistServe and vLLM at the same SLA. To ease adoption in ecosystems like vLLM and SGLang, we further build BlitzLoad , a lightweight checkpoint engine that brings BlitzScale ’s data plane to existing serving engines with only a few lines of code changes.
Zitieren
APA
Zhang, D., Liu, Y., Wang, H., Wei, X., Shan, Y., Chen, R., & Chen, H. (2026). BlitzScale: Fast and Live Large Model Autoscaling with O(1) Host Caching. ACM Transactions on Computer Systems. https://doi.org/10.1145/3807606
Vancouver
Zhang D, Liu Y, Wang H, Wei X, Shan Y, Chen R, et al. BlitzScale: Fast and Live Large Model Autoscaling with O(1) Host Caching. ACM Transactions on Computer Systems. 2026. doi:10.1145/3807606
BibTeX
@article{Zhang2026,
author = {Zhang, Dingyan and Liu, Yang and Wang, Haotian and Wei, Xingda and Shan, Yizhou and Chen, Rong and Chen, Haibo},
title = {BlitzScale: Fast and Live Large Model Autoscaling with O(1) Host Caching},
journal = {ACM Transactions on Computer Systems},
year = {2026},
doi = {10.1145/3807606}
}
EUVIMED