EUVIMEDEUVIMED entdecken
Crossref

StreamOps: Cloud-Native Runtime Management for Streaming Services in ByteDance

Yancan Mao, Zhanghao Chen, Yifan Zhang, Meng Wang, Yong Fang, Guanghui Zhang, Rui Shi, Richard T. B. Ma

Proceedings of the VLDB Endowment · 2023-08 · 16 (12) · 3501-3514

DOI: 10.14778/3611540.3611543 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the VLDB Endowment, 2023-08. Verfasst von Yancan Mao und 7 weiteren Autor:innen. Die Arbeit wurde bisher 18-mal zitiert (Crossref) und nennt 43 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Stream processing is widely used for real-time data processing and decision-making, leading to tens of thousands of streaming jobs deployed in ByteDance cloud. Since those streaming jobs usually run for several days or longer and the input workloads vary over time, they usually face diverse runtime issues such as processing lag and varying failures. This requires runtime management to resolve such runtime issues automatically. However, designing a runtime management service on the ByteDance scale is challenging. In particular, the service has to concurrently manage cluster-wide streaming jobs in a scalable and extensible manner. Furthermore, it should also be able to manage diverse streaming jobs effectively. To this end, we propose StreamOps to enable cloud-native runtime management for streaming jobs in ByteDance. StreamOps has three main designs to address the challenges. 1) To allow for scalability, StreamOps is running as a standalone lightweight control plane to manage cluster-wide streaming jobs. 2) To enable extensible runtime management, StreamOps abstracts control policies to identify and resolve runtime issues. New control policies can be implemented with a detect-diagnose-resolve programming paradigm. Each control policy is also configurable for different streaming jobs according to the performance requirements. 3) To mitigate processing lag and handling failures effectively, StreamOps features three control policies, i.e., auto-scaler, straggler detector, and job doctor, that are inspired by state-of-the-art research and production experiences at ByteDance. In this paper, we introduce the design decisions we made and the experiences we learned from building StreamOps. We evaluate StreamOps in our production environment, and the experiment results have further validated our system design.

Zitieren

APA

Mao, Y., Chen, Z., Zhang, Y., Wang, M., Fang, Y., Zhang, G., Shi, R., & Ma, R. T. B. (2023). StreamOps: Cloud-Native Runtime Management for Streaming Services in ByteDance. Proceedings of the VLDB Endowment, 16(12), 3501-3514. https://doi.org/10.14778/3611540.3611543

Vancouver

Mao Y, Chen Z, Zhang Y, Wang M, Fang Y, Zhang G, et al. StreamOps: Cloud-Native Runtime Management for Streaming Services in ByteDance. Proceedings of the VLDB Endowment. 2023;16(12):3501-3514. doi:10.14778/3611540.3611543

BibTeX
@article{Mao2023,
  author = {Mao, Yancan and Chen, Zhanghao and Zhang, Yifan and Wang, Meng and Fang, Yong and Zhang, Guanghui and Shi, Rui and Ma, Richard T. B.},
  title = {StreamOps: Cloud-Native Runtime Management for Streaming Services in ByteDance},
  journal = {Proceedings of the VLDB Endowment},
  year = {2023},
  volume = {16},
  number = {12},
  pages = {3501--3514},
  doi = {10.14778/3611540.3611543}
}
Kostenlos starten →