SpotDAG: An RL-Based Algorithm for DAG Workflow Scheduling in Heterogeneous Cloud Environments
IEEE Transactions on Services Computing · 2024-09 · 17 (5) · 2904-2917
DOI: 10.1109/tsc.2024.3422828 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Services Computing, 2024-09. Verfasst von Liduo Lin und 2 weiteren Autor:innen. Die Arbeit wurde bisher 32-mal zitiert (Crossref) und nennt 57 Literaturangaben. Gefördert von: National Key R&D Program of China, Natural Science Foundation of Shandong Province.
Abstract (Originalsprache)
As increasingly complex functions are implemented in applications, directed acyclic graphs (DAGs) are widely used to model the inter-dependencies between individual functions. Cloud-based data processing platforms need to consider the complex topology of DAGs and arbitrary deadlines given by users for job scheduling, leading to an NP-hard decision-making problem. Leveraging spot instances in data processing platforms can achieve significant cost savings, but the unpredictable interruption of spot instances makes the problem of VM scaling and job scheduling more difficult. In this paper, a Reinforcement Learning (RL) based approach called SpotDAG is proposed to solve the auto-scaling problem for jobs modeled as DAGs on a data processing platform where spot instances are introduced. SpotDAG makes cluster scaling and job scheduling decisions at the same time by mapping its output to several meta-policies. This paper introduces the self-attention mechanism for feature extraction to help the intelligent agent learn faster. A mask layer after the output of the proposed RL-based algorithm circumvents illegal actions to ensure that a job is completed by its deadline. Extensive experimental results show that the proposed approach can significantly reduce the cost of instances for data processing platforms while ensuring that jobs are completed in time.
Förderung
- National Key R&D Program of China
- Natural Science Foundation of Shandong Province
Zitieren
APA
Lin, L., Pan, L., & Liu, S. (2024). SpotDAG: An RL-Based Algorithm for DAG Workflow Scheduling in Heterogeneous Cloud Environments. IEEE Transactions on Services Computing, 17(5), 2904-2917. https://doi.org/10.1109/tsc.2024.3422828
Vancouver
Lin L, Pan L, Liu S. SpotDAG: An RL-Based Algorithm for DAG Workflow Scheduling in Heterogeneous Cloud Environments. IEEE Transactions on Services Computing. 2024;17(5):2904-2917. doi:10.1109/tsc.2024.3422828
BibTeX
@article{Lin2024,
author = {Lin, Liduo and Pan, Li and Liu, Shijun},
title = {SpotDAG: An RL-Based Algorithm for DAG Workflow Scheduling in Heterogeneous Cloud Environments},
journal = {IEEE Transactions on Services Computing},
year = {2024},
volume = {17},
number = {5},
pages = {2904--2917},
doi = {10.1109/tsc.2024.3422828}
}
EUVIMED