A Communication Efficient ADMM-based Distributed Algorithm Using Two-Dimensional Torus Grouping AllReduce
Data Science and Engineering · 2023-03 · 8 (1) · 61-72
DOI: 10.1007/s41019-022-00202-7 ↗Kurz gefasst
Fachartikel in der Zeitschrift Data Science and Engineering, online erschienen am 02.01.2023. Verfasst von Guozheng Wang und 3 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 19 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Large-scale distributed training mainly consists of sub-model parallel training and parameter synchronization. With the expansion of training workers, the efficiency of parameter synchronization will be affected. To tackle this problem, we first propose 2D-TGA, a g rouping A llReduce method based on the two-dimensional t orus topology. This method synchronizes the model parameters by grouping and makes full use of bandwidth. Secondly, we propose a distributed algorithm, 2D-TGA-ADMM, which combines the 2D-TGA with the alternating direction method of multipliers (ADMM). It focuses on sub-model training and reduces the wait time among workers in the synchronization process. Finally, experimental results on the Tianhe-2 supercomputing platform show that compared with the $${\mathtt {MPI\_Allreduce}}$$ MPI _ Allreduce , the 2D-TGA could shorten the synchronization wait time by $$33\%$$ 33 % .
Zitieren
APA
Wang, G., Lei, Y., Zhang, Z., & Peng, C. (2023). A Communication Efficient ADMM-based Distributed Algorithm Using Two-Dimensional Torus Grouping AllReduce. Data Science and Engineering, 8(1), 61-72. https://doi.org/10.1007/s41019-022-00202-7
Vancouver
Wang G, Lei Y, Zhang Z, Peng C. A Communication Efficient ADMM-based Distributed Algorithm Using Two-Dimensional Torus Grouping AllReduce. Data Science and Engineering. 2023;8(1):61-72. doi:10.1007/s41019-022-00202-7
BibTeX
@article{Wang2023,
author = {Wang, Guozheng and Lei, Yongmei and Zhang, Zeyu and Peng, Cunlu},
title = {A Communication Efficient ADMM-based Distributed Algorithm Using Two-Dimensional Torus Grouping AllReduce},
journal = {Data Science and Engineering},
year = {2023},
volume = {8},
number = {1},
pages = {61--72},
doi = {10.1007/s41019-022-00202-7}
}
EUVIMED