EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

A Communication Efficient ADMM-based Distributed Algorithm Using Two-Dimensional Torus Grouping AllReduce

Guozheng Wang, Yongmei Lei, Zeyu Zhang, Cunlu Peng

Data Science and Engineering · 2023-03 · 8 (1) · 61-72

DOI: 10.1007/s41019-022-00202-7 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Data Science and Engineering, online erschienen am 02.01.2023. Verfasst von Guozheng Wang und 3 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 19 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large-scale distributed training mainly consists of sub-model parallel training and parameter synchronization. With the expansion of training workers, the efficiency of parameter synchronization will be affected. To tackle this problem, we first propose 2D-TGA, a g rouping A llReduce method based on the two-dimensional t orus topology. This method synchronizes the model parameters by grouping and makes full use of bandwidth. Secondly, we propose a distributed algorithm, 2D-TGA-ADMM, which combines the 2D-TGA with the alternating direction method of multipliers (ADMM). It focuses on sub-model training and reduces the wait time among workers in the synchronization process. Finally, experimental results on the Tianhe-2 supercomputing platform show that compared with the $${\mathtt {MPI\_Allreduce}}$$ MPI _ Allreduce , the 2D-TGA could shorten the synchronization wait time by $$33\%$$ 33 % .

Zitieren

APA

Wang, G., Lei, Y., Zhang, Z., & Peng, C. (2023). A Communication Efficient ADMM-based Distributed Algorithm Using Two-Dimensional Torus Grouping AllReduce. Data Science and Engineering, 8(1), 61-72. https://doi.org/10.1007/s41019-022-00202-7

Vancouver

Wang G, Lei Y, Zhang Z, Peng C. A Communication Efficient ADMM-based Distributed Algorithm Using Two-Dimensional Torus Grouping AllReduce. Data Science and Engineering. 2023;8(1):61-72. doi:10.1007/s41019-022-00202-7

BibTeX
@article{Wang2023,
  author = {Wang, Guozheng and Lei, Yongmei and Zhang, Zeyu and Peng, Cunlu},
  title = {A Communication Efficient ADMM-based Distributed Algorithm Using Two-Dimensional Torus Grouping AllReduce},
  journal = {Data Science and Engineering},
  year = {2023},
  volume = {8},
  number = {1},
  pages = {61--72},
  doi = {10.1007/s41019-022-00202-7}
}
Kostenlos starten →