EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Adaptive memory reservation strategy for heavy workloads in the Spark environment

Bohan Li, Xin He, Junyang Yu, Guanghui Wang, Yixin Song, Shunjie Pan, Hangyu Gu

PeerJ Computer Science · 13.11.2024 · 10 · e2460

DOI: 10.7717/peerj-cs.2460 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PeerJ Computer Science, online erschienen am 13.11.2024. Verfasst von Bohan Li und 6 weiteren Autor:innen. Beteiligte Einrichtung: School of Software, Henan University, Kaifeng, Henan Province, China. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 45 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Major Science and Technology Project of Henan Province, Henan Province Science and Technology Research Project, Key Scientific Research Project of Colleges and Universities in Henan Province, China.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The rise of the Internet of Things (IoT) and Industry 2.0 has spurred a growing need for extensive data computing, and Spark emerged as a promising Big Data platform, attributed to its distributed in-memory computing capabilities. However, practical heavy workloads often lead to memory bottleneck issues in the Spark platform. This results in resilient distributed datasets (RDD) eviction and, in extreme cases, violent memory contentions, causing a significant degradation in Spark computational efficiency. To tackle this issue, we propose an adaptive memory reservation (AMR) strategy in this article, specifically designed for heavy workloads in the Spark environment. Specifically, we model optimal task parallelism by minimizing the disparity between the number of tasks completed without blocking and the number completed in regular rounds. Optimal memory for task parallelism is determined to establish an efficient execution memory space for computational parallelism. Subsequently, through adaptive execution memory reservation and dynamic adjustments, such as compression or expansion based on task progress, the strategy ensures dynamic task parallelism in the Spark parallel computing process. Considering the cost of RDD cache location and real-time memory space usage, we select suitable storage locations for different RDD types to alleviate execution memory pressure. Finally, we conduct extensive laboratory experiments to validate the effectiveness of AMR. Results indicate that, compared to existing memory management solutions, AMR reduces the execution time by approximately 46.8%.

Schlagwörter

Stichwörter der Autor:innen

Adaptive memory reservationSpark environmentStorage location selectionTask parallelism

Autor:innen und Institutionen

  • School of Software, Henan University, Kaifeng, Henan Province, China.
  • School of Computer Science and Engineering, Southeast University, Nanjing, Jiangsu Province, China.

Förderung

  • Major Science and Technology Project of Henan Province
  • Henan Province Science and Technology Research Project
  • Key Scientific Research Project of Colleges and Universities in Henan Province, China
  • Key R&D and Promotion Special Project of Henan Province, China
  • Key Research and Promotion Projects of Henan Province

Zitieren

APA

Li, B., He, X., Yu, J., Wang, G., Song, Y., Pan, S., & Gu, H. (2024). Adaptive memory reservation strategy for heavy workloads in the Spark environment. PeerJ Computer Science, 10, e2460. https://doi.org/10.7717/peerj-cs.2460

Vancouver

Li B, He X, Yu J, Wang G, Song Y, Pan S, et al. Adaptive memory reservation strategy for heavy workloads in the Spark environment. PeerJ Computer Science. 2024;10:e2460. doi:10.7717/peerj-cs.2460

BibTeX
@article{Li2024,
  author = {Li, Bohan and He, Xin and Yu, Junyang and Wang, Guanghui and Song, Yixin and Pan, Shunjie and Gu, Hangyu},
  title = {Adaptive memory reservation strategy for heavy workloads in the Spark environment},
  journal = {PeerJ Computer Science},
  year = {2024},
  volume = {10},
  pages = {e2460},
  doi = {10.7717/peerj-cs.2460}
}
Kostenlos starten →