EUVIMEDEUVIMED entdecken
Crossref

RCMoE: A Communication-Efficient Random Compression Framework for Resource-Constrained Mixture-of-Experts Training

Donglei Wu, Xiao Cai, Jinglei Tan, Jinda Jia, Guangming Tan, Dingwen Tao, Wen Xia, Zhihong Tian

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (32) · 26876-26884

DOI: 10.1609/aaai.v40i32.39899 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Donglei Wu und 7 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Mixture-of-Experts (MoE) architecture with experts parallelism scales LLMs efficiently by activating only a subset of experts per input, avoiding proportional training costs. However, the intensive and heterogeneous communication substantially hinders the efficiency and scalability of MoE training in the resource-constrained scenario. Existing communication compression techniques fall short in MoE training due to: (i) Intensive training amplifies compression overhead, compromising training efficiency; (ii) Accumulated compression errors propagate through the network, degrading training quality. In this paper, we propose RCMoE, a communication-efficient Random Compression framework for MoE training with two core modules: (1) Local-Stochastic Quantization compresses the all-to-all communication by stochastically quantizing each row of the expert's intermediate computing results in parallel, effectively improving the compression efficiency and reducing compression error; (2) Probabilistic Thresholding Sparsification compresses the all-reduce communication by probabilistically sampling large gradients at high probability, thereby reducing the computational complexity and maintaining the convergence efficiency. Experiments on four typical MoE training tasks prove that RCMoE achieves higher 5.9x-8.1x total communication compression ratios and 1.3x-10.1x training speedup compared with the state-of-the-art compression techniques while maintaining the MoE training accuracy.

Zitieren

APA

Wu, D., Cai, X., Tan, J., Jia, J., Tan, G., Tao, D., Xia, W., & Tian, Z. (2026). RCMoE: A Communication-Efficient Random Compression Framework for Resource-Constrained Mixture-of-Experts Training. Proceedings of the AAAI Conference on Artificial Intelligence, 40(32), 26876-26884. https://doi.org/10.1609/aaai.v40i32.39899

Vancouver

Wu D, Cai X, Tan J, Jia J, Tan G, Tao D, et al. RCMoE: A Communication-Efficient Random Compression Framework for Resource-Constrained Mixture-of-Experts Training. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(32):26876-26884. doi:10.1609/aaai.v40i32.39899

BibTeX
@article{Wu2026,
  author = {Wu, Donglei and Cai, Xiao and Tan, Jinglei and Jia, Jinda and Tan, Guangming and Tao, Dingwen and Xia, Wen and Tian, Zhihong},
  title = {RCMoE: A Communication-Efficient Random Compression Framework for Resource-Constrained Mixture-of-Experts Training},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {32},
  pages = {26876--26884},
  doi = {10.1609/aaai.v40i32.39899}
}
Kostenlos starten →