EUVIMEDEUVIMED entdecken
Crossref

MPDA: A Massively Parallel Learning and Dependency-Aware Scheduling Algorithm for Data Processing Clusters

Qing Li, Xingchi Chen, Jiawei Guo, Fa Zhu, Achyut Shankar, Fayez Alqahtani, Kamalakanta Muduli, Bo Yi, Yong Jiang

IEEE Transactions on Services Computing · 2025-09 · 18 (5) · 3038-3051

DOI: 10.1109/tsc.2025.3592427 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Services Computing, 2025-09. Verfasst von Qing Li und 8 weiteren Autor:innen. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 56 Literaturangaben. Gefördert von: Department of Strategic and Advanced Interdisciplinary Research of Pengcheng Laboratory, Ongoing Research Funding Program, King Saud University.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In the era of large-scale machine learning, large-scale clusters are extensively used for data processing jobs. However, the state-of-the-art heuristic-based and Deep Reinforcement Learning (DRL) based job scheduling mechanisms are facing challenges such as slow training speed and under-exploitation of jobs’ complex dependencies. We propose MPDA, a Massively Parallel learning and Dependency-Aware scheduling algorithm, consisting of a fast-training mechanism and a novel dependency-aware policy network, GATNetwork, to address these two challenges respectively. The fast-training mechanism is a two-level massively parallel training method that can significantly accelerate the training process and maximally utilize the resources of the cluster. Additionally, its decoupled learning and interacting design enables hybrid-workload training for MPDA, which guarantees the generalization and robustness of MPDA. The GATNetwork exploits the dependencies among stages/jobs using Graph Attention Network (GAT) and Long Short-Term Memory (LSTM) networks to improve the performance of the scheduling policy. The experiments show that MPDA accelerates the training speed by one to two orders of magnitude and achieves better scheduling performance, i.e., lower average job completion time, compared with existing scheduling algorithms.

Förderung

  • Department of Strategic and Advanced Interdisciplinary Research of Pengcheng Laboratory
  • Ongoing Research Funding Program, King Saud University

Zitieren

APA

Li, Q., Chen, X., Guo, J., Zhu, F., Shankar, A., Alqahtani, F., Muduli, K., Yi, B., & Jiang, Y. (2025). MPDA: A Massively Parallel Learning and Dependency-Aware Scheduling Algorithm for Data Processing Clusters. IEEE Transactions on Services Computing, 18(5), 3038-3051. https://doi.org/10.1109/tsc.2025.3592427

Vancouver

Li Q, Chen X, Guo J, Zhu F, Shankar A, Alqahtani F, et al. MPDA: A Massively Parallel Learning and Dependency-Aware Scheduling Algorithm for Data Processing Clusters. IEEE Transactions on Services Computing. 2025;18(5):3038-3051. doi:10.1109/tsc.2025.3592427

BibTeX
@article{Li2025,
  author = {Li, Qing and Chen, Xingchi and Guo, Jiawei and Zhu, Fa and Shankar, Achyut and Alqahtani, Fayez and Muduli, Kamalakanta and Yi, Bo and Jiang, Yong},
  title = {MPDA: A Massively Parallel Learning and Dependency-Aware Scheduling Algorithm for Data Processing Clusters},
  journal = {IEEE Transactions on Services Computing},
  year = {2025},
  volume = {18},
  number = {5},
  pages = {3038--3051},
  doi = {10.1109/tsc.2025.3592427}
}
Kostenlos starten →