EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Collective Communication Performance Evaluation for Distributed Deep Learning Training

Sookwang Lee, Jaehwan Lee

Applied Sciences · 12.06.2024 · 14 (12) · 5100

DOI: 10.3390/app14125100 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 12.06.2024. Verfasst von Sookwang Lee und 1 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 31 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Research Foundation of Korea.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In distributed deep learning, the improper use of the collective communication library can lead to a decline in deep learning performance due to increased communication time. Representative collective communication libraries such as MPI, GLOO, and NCCL exhibit varying performance based on server environment and communication architecture. In this study, we investigate three key aspects to evaluate the performance of the collective communication libraries in a distributed deep learning setting in an intra-node environment. First, we conduct a comparison and analysis of collective communication library performance within common distributed deep learning architectures, such as parameter servers and ring all-reduce methods. Second, we evaluate the performance of these libraries in different environments, including various container platforms and bare metal setups, considering the scalability and flexibility advantages offered by cloud virtualization. Last, to ensure practicality, we assess the libraries’ performance in a Linux shell and within the PyTorch framework. In the cross-docker virtualization environment, NCCL shows up to 213% higher latency compared to single docker, while GLOO exhibits 36% lower latency in single docker than in cross docker, and NCCL achieves up to 345% lower execution time in all-reduce operations compared to other libraries (MPI and GLOO). These findings will inform the selection of an appropriate collective communication library for designing effective distributed deep learning environments.

Förderung

  • National Research Foundation of Korea

Zitieren

APA

Lee, S., & Lee, J. (2024). Collective Communication Performance Evaluation for Distributed Deep Learning Training. Applied Sciences, 14(12), 5100. https://doi.org/10.3390/app14125100

Vancouver

Lee S, Lee J. Collective Communication Performance Evaluation for Distributed Deep Learning Training. Applied Sciences. 2024;14(12):5100. doi:10.3390/app14125100

BibTeX
@article{Lee2024,
  author = {Lee, Sookwang and Lee, Jaehwan},
  title = {Collective Communication Performance Evaluation for Distributed Deep Learning Training},
  journal = {Applied Sciences},
  year = {2024},
  volume = {14},
  number = {12},
  pages = {5100},
  doi = {10.3390/app14125100}
}
Kostenlos starten →