EUVIMEDEUVIMED entdecken
Crossref

PipeDAP: An Efficient Communication Framework for Scheduling Decoupled All-Reduce Primitives in Distributed DNN Training

Yunqi Gao, Bing Hu, Mahdi Boloursaz Mashhadi, Wei Wang, Rahim Tafazolli, Mérouane Debbah

IEEE Transactions on Emerging Topics in Computing · 2025-07 · 13 (3) · 1170-1184

DOI: 10.1109/tetc.2025.3573522 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Emerging Topics in Computing, 2025-07. Verfasst von Yunqi Gao und 5 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 45 Literaturangaben. Gefördert von: National Key Research and Development Project, Zhejiang Provincial Natural Science Foundation of China, China Scholarship Council.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Communication scheduling effectively improves the scalability of distributed deep learning by overlapping computation and communication tasks during training. However, existing communication scheduling frameworks based on tensor partitioning suffer from two fundamental issues: (1) partitioning schemes at the data volume level introduce extensive startup overheads leading to higher energy consumption, and (2) partitioning schemes at the communication primitive level do not provide optimal scheduling resulting in longer training time. In this article, we propose an efficient communication mechanism, namely PipeDAP, which schedules decoupled all-reduce operations in a near-optimal order to minimize the time and energy consumption of training DNN models. We build the mathematical model for PipeDAP and derive the near-optimal scheduling order of the reduce-scatter and all-gather operations. Meanwhile, we leverage simultaneous communication of reduce-scatter and all-gather operations to further reduce the startup overheads. We implement the PipeDAP architecture on PyTorch framework, and apply it for distributed training of benchmark DNN models. Experimental results on two GPU clusters demonstrate that PipeDAP achieves up to 1.82x speedup and saves up to 45.4% of energy consumption compared to the state-of-the-art communication scheduling frameworks.

Förderung

  • National Key Research and Development Project
  • Zhejiang Provincial Natural Science Foundation of China
  • China Scholarship Council

Zitieren

APA

Gao, Y., Hu, B., Mashhadi, M. B., Wang, W., Tafazolli, R., & Debbah, M. (2025). PipeDAP: An Efficient Communication Framework for Scheduling Decoupled All-Reduce Primitives in Distributed DNN Training. IEEE Transactions on Emerging Topics in Computing, 13(3), 1170-1184. https://doi.org/10.1109/tetc.2025.3573522

Vancouver

Gao Y, Hu B, Mashhadi MB, Wang W, Tafazolli R, Debbah M. PipeDAP: An Efficient Communication Framework for Scheduling Decoupled All-Reduce Primitives in Distributed DNN Training. IEEE Transactions on Emerging Topics in Computing. 2025;13(3):1170-1184. doi:10.1109/tetc.2025.3573522

BibTeX
@article{Gao2025,
  author = {Gao, Yunqi and Hu, Bing and Mashhadi, Mahdi Boloursaz and Wang, Wei and Tafazolli, Rahim and Debbah, Mérouane},
  title = {PipeDAP: An Efficient Communication Framework for Scheduling Decoupled All-Reduce Primitives in Distributed DNN Training},
  journal = {IEEE Transactions on Emerging Topics in Computing},
  year = {2025},
  volume = {13},
  number = {3},
  pages = {1170--1184},
  doi = {10.1109/tetc.2025.3573522}
}
Kostenlos starten →