EUVIMEDEUVIMED entdecken
Crossref

Towards Resource Efficiency: Practical Insights into Large-Scale Spark Workloads at ByteDance

Yixin Wu, Xiuqi Huang, Zhongjia Wei, Hang Cheng, Chaohui Xin, Zuzhi Chen, Binbin Chen, Yufei Wu, Hao Wang, Tieying Zhang, Rui Shi, Xiaofeng Gao und 3 weitere

Proceedings of the VLDB Endowment · 2024-08 · 17 (12) · 3759-3771

DOI: 10.14778/3685800.3685804 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the VLDB Endowment, online erschienen am 08.11.2024. Verfasst von Yixin Wu und 14 weiteren Autor:innen. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 45 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

At ByteDance, where we execute over a million Spark jobs and handle 500PB of shuffled data daily, ensuring resource efficiency is paramount for cost savings. However, achieving optimization of resource efficiency in large-scale production environments poses significant challenges. Drawing from our practical experiences, we have identified three key issues critical to addressing resource efficiency in real-world production settings: 1 slow I/Os leading to excessive CPU and memory idleness, 2 coarse-grained resource control causing wastage, and 3 sub-optimal job configurations resulting in low utilization. To tackle these issues, we propose a resource efficiency governance framework for Spark workloads. Specifically, 1 we devise the multi-mechanism shuffle services, including Enhanced External Shuffle Service (ESS) and Cloud Shuffle Service (CSS), where CSS employs a push-based approach to enhance I/O efficiency through sequential reading. 2 We modify the Spark configuration parameter protocol, allowing for fine-grained resource control by introducing several new parameters such as milliCores and memoryBurst, as well as supporting operators with additional spill modes. 3 We design a two-stage configuration autotuning method, comprising rule-based and algorithm-based tuning, providing more reliable Spark configuration optimizations. By deploying these techniques on millions of Spark jobs in production over the last two years, we have achieved over 22% CPU utilization increase, 5% memory utilization increase, and 10% shuffle block time ratio decrease, effectively saving millions of CPU cores and petabytes of memory daily.

Zitieren

APA

Wu, Y., Huang, X., Wei, Z., Cheng, H., Xin, C., Chen, Z., Chen, B., Wu, Y., Wang, H., Zhang, T., Shi, R., Gao, X., Liang, Y., Zhao, P., & Chen, G. (2024). Towards Resource Efficiency: Practical Insights into Large-Scale Spark Workloads at ByteDance. Proceedings of the VLDB Endowment, 17(12), 3759-3771. https://doi.org/10.14778/3685800.3685804

Vancouver

Wu Y, Huang X, Wei Z, Cheng H, Xin C, Chen Z, et al. Towards Resource Efficiency: Practical Insights into Large-Scale Spark Workloads at ByteDance. Proceedings of the VLDB Endowment. 2024;17(12):3759-3771. doi:10.14778/3685800.3685804

BibTeX
@article{Wu2024,
  author = {Wu, Yixin and Huang, Xiuqi and Wei, Zhongjia and Cheng, Hang and Xin, Chaohui and Chen, Zuzhi and Chen, Binbin and Wu, Yufei and Wang, Hao and Zhang, Tieying and Shi, Rui and Gao, Xiaofeng and Liang, Yuming and Zhao, Pengwei and Chen, Guihai},
  title = {Towards Resource Efficiency: Practical Insights into Large-Scale Spark Workloads at ByteDance},
  journal = {Proceedings of the VLDB Endowment},
  year = {2024},
  volume = {17},
  number = {12},
  pages = {3759--3771},
  doi = {10.14778/3685800.3685804}
}
Kostenlos starten →