EUVIMEDEUVIMED entdecken
Crossref

PipeCo: Pipelining Cold Start of Deep Learning Inference Services on Serverless Platforms

Jiaang Duan, Shiyou Qian, Hanwen Hu, Dingyu Yang, Jian Cao, Guangtao Xue

Proceedings of the ACM on Measurement and Analysis of Computing Systems · 27.05.2025 · 9 (2) · 1-23

DOI: 10.1145/3727125 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the ACM on Measurement and Analysis of Computing Systems, online erschienen am 03.06.2025. Verfasst von Jiaang Duan und 5 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 52 Literaturangaben. Gefördert von: Alibaba Group through Alibaba Innovative Research Program.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The fusion of serverless computing and deep learning (DL) has led to serverless inference, offering a promising approach for developing and deploying scalable and cost-efficient deep learning inference services (DLISs). However, the challenge of cold start presents a significant obstacle for DLISs, where DL model size greatly impacts latency. Existing studies mitigate cold starts by extending keep-alive times, which unfortunately leads to decreased resource utilization efficiency. To address this issue, we introduce PipeCo, a system designed to alleviate DLIS cold start. The core concept of PipeCo is to achieve the miniaturization and pipelining of DLIS cold start. Firstly, PipeCo utilizes a vertical partitioning approach to divide each DLIS into multiple slices, prewarming slices in a sequential and overlapping manner to decrease the overall cold-start latency. Secondly, PipeCo employs an attention-based prediction mechanism to estimate periodic patterns in requests and idle containers for scheduling slices. Thirdly, PipeCo incorporates a similarity-based container matcher for the reuse of idle containers. We implemented a prototype of PipeCo on the OpenFaaS platform and conducted extensive experiments using three real-world DLIS repositories. The results demonstrate that PipeCo effectively decreases end-to-end (E2E) latency by up to 62.67% on CPU and 58.81% on GPU clusters and reduces the overall resource usage by 65.31% compared to five state-of-the-art baselines.

Förderung

  • Alibaba Group through Alibaba Innovative Research Program

Zitieren

APA

Duan, J., Qian, S., Hu, H., Yang, D., Cao, J., & Xue, G. (2025). PipeCo: Pipelining Cold Start of Deep Learning Inference Services on Serverless Platforms. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 9(2), 1-23. https://doi.org/10.1145/3727125

Vancouver

Duan J, Qian S, Hu H, Yang D, Cao J, Xue G. PipeCo: Pipelining Cold Start of Deep Learning Inference Services on Serverless Platforms. Proceedings of the ACM on Measurement and Analysis of Computing Systems. 2025;9(2):1-23. doi:10.1145/3727125

BibTeX
@article{Duan2025,
  author = {Duan, Jiaang and Qian, Shiyou and Hu, Hanwen and Yang, Dingyu and Cao, Jian and Xue, Guangtao},
  title = {PipeCo: Pipelining Cold Start of Deep Learning Inference Services on Serverless Platforms},
  journal = {Proceedings of the ACM on Measurement and Analysis of Computing Systems},
  year = {2025},
  volume = {9},
  number = {2},
  pages = {1--23},
  doi = {10.1145/3727125}
}
Kostenlos starten →