Dynamic Load Balancing and Communication Optimization for Data-Parallel Training on Heterogeneous Clusters
Aerospace · 03.11.2026 · 13 (11) · 1021
DOI: 10.3390/aerospace13111021 ↗Kurz gefasst
Fachartikel in der Zeitschrift Aerospace, online erschienen am 03.11.2026. Verfasst von Ruiming Wen und 3 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
The growing adoption of artificial intelligence in space communications has imposed increasingly demanding efficiency requirements on deep learning training for tasks such as intelligent signal processing and resource allocation. However, performance disparities and workload fluctuations across heterogeneous nodes cause synchronization delays, while gradient aggregation and redundant parameter updates further increase training overhead. This paper proposes a data-parallel training method that combines dynamic load balancing with communication optimization. The Iteration-Level Dynamic Batch Adjustment algorithm (IDAT) estimates node processing rates online and dynamically assigns local batch sizes while maintaining a fixed global batch size, with a time-variation threshold, integer adjustment, and GPU memory constraints used to control overhead. The communication optimization algorithm (MRA) integrates Reduce-Scatter, sharded parameter updates, and All-Gather into a unified workflow to reduce communication startup and redundant update costs. Experiments using ResNet-50, AlexNet, and VGG-16 on a cluster of three servers and five NVIDIA V100-series GPUs show that IDAT achieves a 1.03×–1.04× speedup over the baseline, while the combined IDAT–MRA scheme achieves a 1.41×–1.82× speedup over the native PyTorch DDP 1.12.1 and Horovod 0.28.1 configurations, with an absolute accuracy difference of no more than 0.02 percentage points. These results demonstrate the effectiveness of the proposed framework in improving synchronous data-parallel training efficiency on the evaluated heterogeneous cluster and indicate its potential applicability to the training of intelligent models for space communication systems.
Zitieren
APA
Wen, R., Hou, W., Wen, H., & Tian, W. (2026). Dynamic Load Balancing and Communication Optimization for Data-Parallel Training on Heterogeneous Clusters. Aerospace, 13(11), 1021. https://doi.org/10.3390/aerospace13111021
Vancouver
Wen R, Hou W, Wen H, Tian W. Dynamic Load Balancing and Communication Optimization for Data-Parallel Training on Heterogeneous Clusters. Aerospace. 2026;13(11):1021. doi:10.3390/aerospace13111021
BibTeX
@article{Wen2026,
author = {Wen, Ruiming and Hou, Wenjing and Wen, Hongchen and Tian, Wenhong},
title = {Dynamic Load Balancing and Communication Optimization for Data-Parallel Training on Heterogeneous Clusters},
journal = {Aerospace},
year = {2026},
volume = {13},
number = {11},
pages = {1021},
doi = {10.3390/aerospace13111021}
}
EUVIMED