SDPipe: A Semi-Decentralized Framework for Heterogeneity-Aware Pipeline-parallel Training
Proceedings of the VLDB Endowment · 2023-05 · 16 (9) · 2354-2363
DOI: 10.14778/3598581.3598604 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the VLDB Endowment, online erschienen am 10.07.2023. Verfasst von Xupeng Miao und 4 weiteren Autor:innen. Die Arbeit wurde bisher 33-mal zitiert (Crossref) und nennt 71 Literaturangaben.
Abstract (Originalsprache)
The increasing size of both deep learning models and training data necessitates the ability to scale out model training through pipeline-parallel training, which combines pipelined model parallelism and data parallelism. However, most of them assume an ideal homogeneous dedicated cluster. As for real cloud clusters, these approaches suffer from the intensive model synchronization overheads due to the dynamic environment heterogeneity. Such a huge challenge leaves the design in a dilemma: either the performance bottleneck of the central parameter server (PS) or severe performance degradation caused by stragglers for decentralized synchronization (like All-Reduce). This approach presents SDPipe, a new semi-decentralized framework to get the best of both worlds, achieving both high heterogeneity tolerance and convergence efficiency in pipeline-parallel training. To provide high performance, we decentralize the communication model synchronization, which accounts for the largest proportion of synchronization overhead. In contrast, we centralize the process of group scheduling, which is lightweight but needs a global view for better performance and convergence speed against heterogeneity. We show via a prototype implementation the significant advantage of SDPipe on performance and scalability, facing different environments.
Zitieren
APA
Miao, X., Shi, Y., Yang, Z., Cui, B., & Jia, Z. (2023). SDPipe: A Semi-Decentralized Framework for Heterogeneity-Aware Pipeline-parallel Training. Proceedings of the VLDB Endowment, 16(9), 2354-2363. https://doi.org/10.14778/3598581.3598604
Vancouver
Miao X, Shi Y, Yang Z, Cui B, Jia Z. SDPipe: A Semi-Decentralized Framework for Heterogeneity-Aware Pipeline-parallel Training. Proceedings of the VLDB Endowment. 2023;16(9):2354-2363. doi:10.14778/3598581.3598604
BibTeX
@article{Miao2023,
author = {Miao, Xupeng and Shi, Yining and Yang, Zhi and Cui, Bin and Jia, Zhihao},
title = {SDPipe: A Semi-Decentralized Framework for Heterogeneity-Aware Pipeline-parallel Training},
journal = {Proceedings of the VLDB Endowment},
year = {2023},
volume = {16},
number = {9},
pages = {2354--2363},
doi = {10.14778/3598581.3598604}
}
EUVIMED