RCMoE: A Communication-Efficient Random Compression Framework for Resource-Constrained Mixture-of-Experts Training
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (32) · 26876-26884
DOI: 10.1609/aaai.v40i32.39899 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Donglei Wu und 7 weiteren Autor:innen.
Abstract (Originalsprache)
Mixture-of-Experts (MoE) architecture with experts parallelism scales LLMs efficiently by activating only a subset of experts per input, avoiding proportional training costs. However, the intensive and heterogeneous communication substantially hinders the efficiency and scalability of MoE training in the resource-constrained scenario. Existing communication compression techniques fall short in MoE training due to: (i) Intensive training amplifies compression overhead, compromising training efficiency; (ii) Accumulated compression errors propagate through the network, degrading training quality. In this paper, we propose RCMoE, a communication-efficient Random Compression framework for MoE training with two core modules: (1) Local-Stochastic Quantization compresses the all-to-all communication by stochastically quantizing each row of the expert's intermediate computing results in parallel, effectively improving the compression efficiency and reducing compression error; (2) Probabilistic Thresholding Sparsification compresses the all-reduce communication by probabilistically sampling large gradients at high probability, thereby reducing the computational complexity and maintaining the convergence efficiency. Experiments on four typical MoE training tasks prove that RCMoE achieves higher 5.9x-8.1x total communication compression ratios and 1.3x-10.1x training speedup compared with the state-of-the-art compression techniques while maintaining the MoE training accuracy.
Zitieren
APA
Wu, D., Cai, X., Tan, J., Jia, J., Tan, G., Tao, D., Xia, W., & Tian, Z. (2026). RCMoE: A Communication-Efficient Random Compression Framework for Resource-Constrained Mixture-of-Experts Training. Proceedings of the AAAI Conference on Artificial Intelligence, 40(32), 26876-26884. https://doi.org/10.1609/aaai.v40i32.39899
Vancouver
Wu D, Cai X, Tan J, Jia J, Tan G, Tao D, et al. RCMoE: A Communication-Efficient Random Compression Framework for Resource-Constrained Mixture-of-Experts Training. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(32):26876-26884. doi:10.1609/aaai.v40i32.39899
BibTeX
@article{Wu2026,
author = {Wu, Donglei and Cai, Xiao and Tan, Jinglei and Jia, Jinda and Tan, Guangming and Tao, Dingwen and Xia, Wen and Tian, Zhihong},
title = {RCMoE: A Communication-Efficient Random Compression Framework for Resource-Constrained Mixture-of-Experts Training},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {32},
pages = {26876--26884},
doi = {10.1609/aaai.v40i32.39899}
}
EUVIMED