A Cost‐Aware and Latency‐Benefit Evaluation‐Based Task Scheduling Optimization Strategy in Apache Spark
Concurrency and Computation: Practice and Experience · 25.10.2025 · 37 (23-24)
DOI: 10.1002/cpe.70244 ↗Kurz gefasst
Fachartikel in der Zeitschrift Concurrency and Computation: Practice and Experience, online erschienen am 03.09.2025. Verfasst von Qingsong Xu und 4 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 31 Literaturangaben. Gefördert von: National Natural Science Foundation of China.
Abstract (Originalsprache)
In the Spark distributed framework, data communication problems (network transfer overhead, network IO bottlenecks) caused by data transfer across nodes/racks are a common cause of performance degradation due to the inconsistency between the task execution location and the data location. Additionally, in heterogeneous environments, Spark's task scheduling strategy cannot fully utilize the advantages of high‐performance nodes. To address the above issues, firstly, this paper proposes a cost‐aware task selection strategy. The strategy models the cost of tasks by considering the impact of data locality and heterogeneous factors on the efficiency of job execution. For scenarios where data locality needs to be reduced for scheduling tasks, the task scheduling problem is transformed into a minimum weighted bipartite graph matching problem, and a greedy matching algorithm is used to solve for the minimum processing cost option. For scenarios that maintain the current data localization level for scheduling tasks, select the task execution with the largest change in task processing cost due to data localization changes. Secondly, the problem is that Spark's delay scheduling algorithm causes resources in the cluster to be in an unnecessary waiting state and reduces cluster resource utilization. In this paper, we propose an adaptive adjustment strategy for delay waiting time based on benefit assessment. This policy improves the resource utilization of the cluster by evaluating the benefit of delay waiting of the scheduler and dynamically adjusting the delay time based on the result of the evaluation. Finally, we implement the proposed strategy in Spark 3.0.0 and evaluate its performance using some representative benchmarks. The experimental results show that, compared with other task scheduling algorithms, the strategy proposed in this paper can effectively improve the execution efficiency of jobs, reduce the execution time of jobs by 15.8%–31.9%, and at the same time reduce the network traffic and improve the CPU utilization.
Förderung
- National Natural Science Foundation of China
Zitieren
APA
Xu, Q., Wang, C., Yu, J., Fei, H., & Ren, X. (2025). A Cost‐Aware and Latency‐Benefit Evaluation‐Based Task Scheduling Optimization Strategy in Apache Spark. Concurrency and Computation: Practice and Experience, 37(23-24). https://doi.org/10.1002/cpe.70244
Vancouver
Xu Q, Wang C, Yu J, Fei H, Ren X. A Cost‐Aware and Latency‐Benefit Evaluation‐Based Task Scheduling Optimization Strategy in Apache Spark. Concurrency and Computation: Practice and Experience. 2025;37(23-24). doi:10.1002/cpe.70244
BibTeX
@article{Xu2025,
author = {Xu, Qingsong and Wang, Congyang and Yu, Junyang and Fei, Haifeng and Ren, Xiaojin},
title = {A Cost‐Aware and Latency‐Benefit Evaluation‐Based Task Scheduling Optimization Strategy in Apache Spark},
journal = {Concurrency and Computation: Practice and Experience},
year = {2025},
volume = {37},
number = {23-24},
doi = {10.1002/cpe.70244}
}
EUVIMED