Enabling Low-Latency, GPU-Efficient Serverless Inference with Model Swapping
ACM Transactions on Architecture and Code Optimization · 30.06.2026 · 23 (2) · 1-27
DOI: 10.1145/3800690 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Architecture and Code Optimization, online erschienen am 25.06.2026. Verfasst von Minchen Yu und 10 weiteren Autor:innen. Die Arbeit nennt 73 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Serverless computing offers a compelling cloud model for online inference services. However, existing serverless platforms lack efficient support for GPUs, hindering their ability to deliver high-performance inference. In this article, we present Torpor , a serverless platform for GPU-efficient, low-latency inference. To enable efficient sharing of a node’s GPUs among numerous inference functions, Torpor maintains models in main memory and dynamically swaps them onto GPUs upon request arrivals (i.e., late binding with model swapping). Torpor uses various techniques, including asynchronous API redirection, GPU runtime sharing, pipelined model execution, and efficient GPU memory management, to minimize latency overhead caused by model swapping. Additionally, we design an interference-aware request scheduling algorithm that utilizes high-speed GPU interconnects to meet latency service-level objectives (SLOs) for individual inference functions. We have implemented Torpor and evaluated its performance in a production environment. Utilizing late binding and model swapping, Torpor can concurrently serve hundreds of inference functions on a worker node with 4 GPUs, while achieving latency performance comparable to native execution, where each model is cached exclusively on a GPU. Pilot deployment in a leading commercial serverless cloud shows that Torpor reduces the GPU provisioning cost by 70% and 65% for users and the platform, respectively.
Zitieren
APA
Yu, M., Wang, A., Wu, B., Liu, Y., Chen, D., Yu, H., Wang, W., Chen, R., Nie, D., Yang, H., & Ding, Y. (2026). Enabling Low-Latency, GPU-Efficient Serverless Inference with Model Swapping. ACM Transactions on Architecture and Code Optimization, 23(2), 1-27. https://doi.org/10.1145/3800690
Vancouver
Yu M, Wang A, Wu B, Liu Y, Chen D, Yu H, et al. Enabling Low-Latency, GPU-Efficient Serverless Inference with Model Swapping. ACM Transactions on Architecture and Code Optimization. 2026;23(2):1-27. doi:10.1145/3800690
BibTeX
@article{Yu2026,
author = {Yu, Minchen and Wang, Ao and Wu, Bohui and Liu, Yuxuan and Chen, Dong and Yu, Haoxuan and Wang, Wei and Chen, Ruichuan and Nie, Dapeng and Yang, Haoran and Ding, Yu},
title = {Enabling Low-Latency, GPU-Efficient Serverless Inference with Model Swapping},
journal = {ACM Transactions on Architecture and Code Optimization},
year = {2026},
volume = {23},
number = {2},
pages = {1--27},
doi = {10.1145/3800690}
}
EUVIMED