Adaptive memory reservation strategy for heavy workloads in the Spark environment
PeerJ Computer Science · 13.11.2024 · 10 · e2460
DOI: 10.7717/peerj-cs.2460 ↗Kurz gefasst
Fachartikel in der Zeitschrift PeerJ Computer Science, online erschienen am 13.11.2024. Verfasst von Bohan Li und 6 weiteren Autor:innen. Beteiligte Einrichtung: School of Software, Henan University, Kaifeng, Henan Province, China. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 45 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Major Science and Technology Project of Henan Province, Henan Province Science and Technology Research Project, Key Scientific Research Project of Colleges and Universities in Henan Province, China.
Abstract (Originalsprache)
The rise of the Internet of Things (IoT) and Industry 2.0 has spurred a growing need for extensive data computing, and Spark emerged as a promising Big Data platform, attributed to its distributed in-memory computing capabilities. However, practical heavy workloads often lead to memory bottleneck issues in the Spark platform. This results in resilient distributed datasets (RDD) eviction and, in extreme cases, violent memory contentions, causing a significant degradation in Spark computational efficiency. To tackle this issue, we propose an adaptive memory reservation (AMR) strategy in this article, specifically designed for heavy workloads in the Spark environment. Specifically, we model optimal task parallelism by minimizing the disparity between the number of tasks completed without blocking and the number completed in regular rounds. Optimal memory for task parallelism is determined to establish an efficient execution memory space for computational parallelism. Subsequently, through adaptive execution memory reservation and dynamic adjustments, such as compression or expansion based on task progress, the strategy ensures dynamic task parallelism in the Spark parallel computing process. Considering the cost of RDD cache location and real-time memory space usage, we select suitable storage locations for different RDD types to alleviate execution memory pressure. Finally, we conduct extensive laboratory experiments to validate the effectiveness of AMR. Results indicate that, compared to existing memory management solutions, AMR reduces the execution time by approximately 46.8%.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- School of Software, Henan University, Kaifeng, Henan Province, China.
- School of Computer Science and Engineering, Southeast University, Nanjing, Jiangsu Province, China.
Förderung
- Major Science and Technology Project of Henan Province
- Henan Province Science and Technology Research Project
- Key Scientific Research Project of Colleges and Universities in Henan Province, China
- Key R&D and Promotion Special Project of Henan Province, China
- Key Research and Promotion Projects of Henan Province
Zitieren
APA
Li, B., He, X., Yu, J., Wang, G., Song, Y., Pan, S., & Gu, H. (2024). Adaptive memory reservation strategy for heavy workloads in the Spark environment. PeerJ Computer Science, 10, e2460. https://doi.org/10.7717/peerj-cs.2460
Vancouver
Li B, He X, Yu J, Wang G, Song Y, Pan S, et al. Adaptive memory reservation strategy for heavy workloads in the Spark environment. PeerJ Computer Science. 2024;10:e2460. doi:10.7717/peerj-cs.2460
BibTeX
@article{Li2024,
author = {Li, Bohan and He, Xin and Yu, Junyang and Wang, Guanghui and Song, Yixin and Pan, Shunjie and Gu, Hangyu},
title = {Adaptive memory reservation strategy for heavy workloads in the Spark environment},
journal = {PeerJ Computer Science},
year = {2024},
volume = {10},
pages = {e2460},
doi = {10.7717/peerj-cs.2460}
}
EUVIMED