MPDA: A Massively Parallel Learning and Dependency-Aware Scheduling Algorithm for Data Processing Clusters
IEEE Transactions on Services Computing · 2025-09 · 18 (5) · 3038-3051
DOI: 10.1109/tsc.2025.3592427 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Services Computing, 2025-09. Verfasst von Qing Li und 8 weiteren Autor:innen. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 56 Literaturangaben. Gefördert von: Department of Strategic and Advanced Interdisciplinary Research of Pengcheng Laboratory, Ongoing Research Funding Program, King Saud University.
Abstract (Originalsprache)
In the era of large-scale machine learning, large-scale clusters are extensively used for data processing jobs. However, the state-of-the-art heuristic-based and Deep Reinforcement Learning (DRL) based job scheduling mechanisms are facing challenges such as slow training speed and under-exploitation of jobs’ complex dependencies. We propose MPDA, a Massively Parallel learning and Dependency-Aware scheduling algorithm, consisting of a fast-training mechanism and a novel dependency-aware policy network, GATNetwork, to address these two challenges respectively. The fast-training mechanism is a two-level massively parallel training method that can significantly accelerate the training process and maximally utilize the resources of the cluster. Additionally, its decoupled learning and interacting design enables hybrid-workload training for MPDA, which guarantees the generalization and robustness of MPDA. The GATNetwork exploits the dependencies among stages/jobs using Graph Attention Network (GAT) and Long Short-Term Memory (LSTM) networks to improve the performance of the scheduling policy. The experiments show that MPDA accelerates the training speed by one to two orders of magnitude and achieves better scheduling performance, i.e., lower average job completion time, compared with existing scheduling algorithms.
Förderung
- Department of Strategic and Advanced Interdisciplinary Research of Pengcheng Laboratory
- Ongoing Research Funding Program, King Saud University
Zitieren
APA
Li, Q., Chen, X., Guo, J., Zhu, F., Shankar, A., Alqahtani, F., Muduli, K., Yi, B., & Jiang, Y. (2025). MPDA: A Massively Parallel Learning and Dependency-Aware Scheduling Algorithm for Data Processing Clusters. IEEE Transactions on Services Computing, 18(5), 3038-3051. https://doi.org/10.1109/tsc.2025.3592427
Vancouver
Li Q, Chen X, Guo J, Zhu F, Shankar A, Alqahtani F, et al. MPDA: A Massively Parallel Learning and Dependency-Aware Scheduling Algorithm for Data Processing Clusters. IEEE Transactions on Services Computing. 2025;18(5):3038-3051. doi:10.1109/tsc.2025.3592427
BibTeX
@article{Li2025,
author = {Li, Qing and Chen, Xingchi and Guo, Jiawei and Zhu, Fa and Shankar, Achyut and Alqahtani, Fayez and Muduli, Kamalakanta and Yi, Bo and Jiang, Yong},
title = {MPDA: A Massively Parallel Learning and Dependency-Aware Scheduling Algorithm for Data Processing Clusters},
journal = {IEEE Transactions on Services Computing},
year = {2025},
volume = {18},
number = {5},
pages = {3038--3051},
doi = {10.1109/tsc.2025.3592427}
}
EUVIMED