PipeCo: Pipelining Cold Start of Deep Learning Inference Services on Serverless Platforms
Proceedings of the ACM on Measurement and Analysis of Computing Systems · 27.05.2025 · 9 (2) · 1-23
DOI: 10.1145/3727125 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the ACM on Measurement and Analysis of Computing Systems, online erschienen am 03.06.2025. Verfasst von Jiaang Duan und 5 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 52 Literaturangaben. Gefördert von: Alibaba Group through Alibaba Innovative Research Program.
Abstract (Originalsprache)
The fusion of serverless computing and deep learning (DL) has led to serverless inference, offering a promising approach for developing and deploying scalable and cost-efficient deep learning inference services (DLISs). However, the challenge of cold start presents a significant obstacle for DLISs, where DL model size greatly impacts latency. Existing studies mitigate cold starts by extending keep-alive times, which unfortunately leads to decreased resource utilization efficiency. To address this issue, we introduce PipeCo, a system designed to alleviate DLIS cold start. The core concept of PipeCo is to achieve the miniaturization and pipelining of DLIS cold start. Firstly, PipeCo utilizes a vertical partitioning approach to divide each DLIS into multiple slices, prewarming slices in a sequential and overlapping manner to decrease the overall cold-start latency. Secondly, PipeCo employs an attention-based prediction mechanism to estimate periodic patterns in requests and idle containers for scheduling slices. Thirdly, PipeCo incorporates a similarity-based container matcher for the reuse of idle containers. We implemented a prototype of PipeCo on the OpenFaaS platform and conducted extensive experiments using three real-world DLIS repositories. The results demonstrate that PipeCo effectively decreases end-to-end (E2E) latency by up to 62.67% on CPU and 58.81% on GPU clusters and reduces the overall resource usage by 65.31% compared to five state-of-the-art baselines.
Förderung
- Alibaba Group through Alibaba Innovative Research Program
Zitieren
APA
Duan, J., Qian, S., Hu, H., Yang, D., Cao, J., & Xue, G. (2025). PipeCo: Pipelining Cold Start of Deep Learning Inference Services on Serverless Platforms. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 9(2), 1-23. https://doi.org/10.1145/3727125
Vancouver
Duan J, Qian S, Hu H, Yang D, Cao J, Xue G. PipeCo: Pipelining Cold Start of Deep Learning Inference Services on Serverless Platforms. Proceedings of the ACM on Measurement and Analysis of Computing Systems. 2025;9(2):1-23. doi:10.1145/3727125
BibTeX
@article{Duan2025,
author = {Duan, Jiaang and Qian, Shiyou and Hu, Hanwen and Yang, Dingyu and Cao, Jian and Xue, Guangtao},
title = {PipeCo: Pipelining Cold Start of Deep Learning Inference Services on Serverless Platforms},
journal = {Proceedings of the ACM on Measurement and Analysis of Computing Systems},
year = {2025},
volume = {9},
number = {2},
pages = {1--23},
doi = {10.1145/3727125}
}
EUVIMED