Towards Optimal Communication Scheduling With Automatic Configuration for Distributed DNN Training
IEEE Transactions on Networking · 2026 · 34 · 4219-4232
DOI: 10.1109/ton.2026.3673659 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Networking, 2026. Verfasst von Jinbin Hu und 7 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 77 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Natural Science Foundation of Hunan Province, Hong Kong RGC TRS.
Abstract (Originalsprache)
ByteScheduler partitions and rearranges tensor transmissions to improve the communication efficiency of distributed Deep Neural Network (DNN) training. The configuration of hyper-parameters (i.e., the partition size and the credit size) is critical to the effectiveness of partitioning and rearrangement. Currently ByteScheduler adopts Bayesian Optimization (BO) to find the optimal configuration for the hyper-parameters beforehand. In practice, however, various runtime factors (such as worker node status and network conditions) change over time, making the statically-determined one-shot configuration result suboptimal for real-world DNN training. To address this problem, in this paper we present a realtime configuration method (called AutoByte) that automatically and timely searches the optimal hyper-parameters as the training systems dynamically change. AutoByte extends the ByteScheduler framework with a meta network, which takes the systems’ runtime statistics as its input, dynamically adjusts the triggering threshold based on system environment characteristics, and outputs predictions for speedups under specific configurations. Evaluation results on various DNN models show that AutoByte can dynamically tune the hyper-parameters with low resource usage, and deliver up to 33.2% higher performance than the best static configuration method on the ByteScheduler framework.
Förderung
- National Natural Science Foundation of China
- Natural Science Foundation of Hunan Province
- Hong Kong RGC TRS
Zitieren
APA
Hu, J., Xu, X., Wang, H., Ma, Y., Chen, X., Zhang, Y., Wang, J., & Chen, K. (2026). Towards Optimal Communication Scheduling With Automatic Configuration for Distributed DNN Training. IEEE Transactions on Networking, 34, 4219-4232. https://doi.org/10.1109/ton.2026.3673659
Vancouver
Hu J, Xu X, Wang H, Ma Y, Chen X, Zhang Y, et al. Towards Optimal Communication Scheduling With Automatic Configuration for Distributed DNN Training. IEEE Transactions on Networking. 2026;34:4219-4232. doi:10.1109/ton.2026.3673659
BibTeX
@article{Hu2026,
author = {Hu, Jinbin and Xu, Xinming and Wang, Hao and Ma, Yiqing and Chen, Xiaolin and Zhang, Yiming and Wang, Jin and Chen, Kai},
title = {Towards Optimal Communication Scheduling With Automatic Configuration for Distributed DNN Training},
journal = {IEEE Transactions on Networking},
year = {2026},
volume = {34},
pages = {4219--4232},
doi = {10.1109/ton.2026.3673659}
}
EUVIMED