EUVIMEDEUVIMED entdecken
Crossref

Congestion-Aware Critical Gradient Scheduling for Distributed Machine Learning in Data Center Networks

Zehua Guo, Jiayu Wang, Sen Liu, Jineng Ren, Yang Xu, Chao Yao

IEEE Transactions on Cloud Computing · 01.07.2023 · 11 (3) · 2296-2311

DOI: 10.1109/tcc.2022.3197350 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Cloud Computing, erschienen am 01.07.2023. Verfasst von Zehua Guo und 5 weiteren Autor:innen. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 42 Literaturangaben. Gefördert von: Key-Area Research and Development Program of Guangdong Province, National Natural Science Foundation of China, Zhejiang Lab Open Research.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Distributed Machine Learning (DML) is proposed not only to accelerate the training of machine learning, but also to solve the inadequate ability for handling a large amount of training data. It adopts multiple computing nodes in data center to collaboratively work in parallel at the cost of high communication overhead. Gradient Compression (GC) is introduced to reduce the communication overhead by reducing the number of synchronized gradients among computing nodes. However, existing GC solutions suffer from varying network congestion. To be specific, when some computing nodes experience high network congestion, their gradient transmission process could be significantly delayed, slowing down the entire training process. To solve the problem, we propose FLASH, a congestion-aware GC solution for DML. FLASH accelerates the training process by jointly considering the iterative approximation of machine learning and dynamic network congestion scenarios. It can maintain good training performance by adaptively adjust and schedule the number of synchronized gradients among computing nodes. We evaluate the effectiveness of FLASH using AlexNet and Resnet18 under different network congestion scenarios. Simulation results show that under the same number of training epochs, FLASH reduces training time 22-71%, maintains good accuracy, and low loss, compared with the existing memory top-K GC solution.

Förderung

  • Key-Area Research and Development Program of Guangdong Province
  • National Natural Science Foundation of China
  • Zhejiang Lab Open Research
  • SongShan Laboratory Fund
  • Beijing Institute of Technology Research Fund Program for Young Scholars
  • China Postdoctoral Science Foundation
  • Fundamental Research Funds for the Central Universities

Zitieren

APA

Guo, Z., Wang, J., Liu, S., Ren, J., Xu, Y., & Yao, C. (2023). Congestion-Aware Critical Gradient Scheduling for Distributed Machine Learning in Data Center Networks. IEEE Transactions on Cloud Computing, 11(3), 2296-2311. https://doi.org/10.1109/tcc.2022.3197350

Vancouver

Guo Z, Wang J, Liu S, Ren J, Xu Y, Yao C. Congestion-Aware Critical Gradient Scheduling for Distributed Machine Learning in Data Center Networks. IEEE Transactions on Cloud Computing. 2023;11(3):2296-2311. doi:10.1109/tcc.2022.3197350

BibTeX
@article{Guo2023,
  author = {Guo, Zehua and Wang, Jiayu and Liu, Sen and Ren, Jineng and Xu, Yang and Yao, Chao},
  title = {Congestion-Aware Critical Gradient Scheduling for Distributed Machine Learning in Data Center Networks},
  journal = {IEEE Transactions on Cloud Computing},
  year = {2023},
  volume = {11},
  number = {3},
  pages = {2296--2311},
  doi = {10.1109/tcc.2022.3197350}
}
Kostenlos starten →