EUVIMEDEUVIMED entdecken
Crossref

SpotDAG: An RL-Based Algorithm for DAG Workflow Scheduling in Heterogeneous Cloud Environments

Liduo Lin, Li Pan, Shijun Liu

IEEE Transactions on Services Computing · 2024-09 · 17 (5) · 2904-2917

DOI: 10.1109/tsc.2024.3422828 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Services Computing, 2024-09. Verfasst von Liduo Lin und 2 weiteren Autor:innen. Die Arbeit wurde bisher 32-mal zitiert (Crossref) und nennt 57 Literaturangaben. Gefördert von: National Key R&D Program of China, Natural Science Foundation of Shandong Province.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

As increasingly complex functions are implemented in applications, directed acyclic graphs (DAGs) are widely used to model the inter-dependencies between individual functions. Cloud-based data processing platforms need to consider the complex topology of DAGs and arbitrary deadlines given by users for job scheduling, leading to an NP-hard decision-making problem. Leveraging spot instances in data processing platforms can achieve significant cost savings, but the unpredictable interruption of spot instances makes the problem of VM scaling and job scheduling more difficult. In this paper, a Reinforcement Learning (RL) based approach called SpotDAG is proposed to solve the auto-scaling problem for jobs modeled as DAGs on a data processing platform where spot instances are introduced. SpotDAG makes cluster scaling and job scheduling decisions at the same time by mapping its output to several meta-policies. This paper introduces the self-attention mechanism for feature extraction to help the intelligent agent learn faster. A mask layer after the output of the proposed RL-based algorithm circumvents illegal actions to ensure that a job is completed by its deadline. Extensive experimental results show that the proposed approach can significantly reduce the cost of instances for data processing platforms while ensuring that jobs are completed in time.

Förderung

  • National Key R&D Program of China
  • Natural Science Foundation of Shandong Province

Zitieren

APA

Lin, L., Pan, L., & Liu, S. (2024). SpotDAG: An RL-Based Algorithm for DAG Workflow Scheduling in Heterogeneous Cloud Environments. IEEE Transactions on Services Computing, 17(5), 2904-2917. https://doi.org/10.1109/tsc.2024.3422828

Vancouver

Lin L, Pan L, Liu S. SpotDAG: An RL-Based Algorithm for DAG Workflow Scheduling in Heterogeneous Cloud Environments. IEEE Transactions on Services Computing. 2024;17(5):2904-2917. doi:10.1109/tsc.2024.3422828

BibTeX
@article{Lin2024,
  author = {Lin, Liduo and Pan, Li and Liu, Shijun},
  title = {SpotDAG: An RL-Based Algorithm for DAG Workflow Scheduling in Heterogeneous Cloud Environments},
  journal = {IEEE Transactions on Services Computing},
  year = {2024},
  volume = {17},
  number = {5},
  pages = {2904--2917},
  doi = {10.1109/tsc.2024.3422828}
}
Kostenlos starten →