EUVIMEDEUVIMED entdecken
Crossref

Efficient Mixture-of-Experts Training With Pipelining Optimization

Ne Wang, Wenxiang Lin, Lin Zhang, Shaohuai Shi, Ruiting Zhou, Bo Li

IEEE Transactions on Computers · 2026-10 · 75 (10) · 3528-3543

DOI: 10.1109/tc.2026.3702749 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Computers, 2026-10. Verfasst von Ne Wang und 5 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 44 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies, Shenzhen Science and Technology Program.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Sparsely activated Mixture-of-Experts (MoE) has emerged as a key technique to expand the size of Transformer-based large language models (LLMs) while maintaining low computational costs. However, MoE layers require to route the input data to distributed devices, incurring significant communication latency. Existing studies have primarily focused on alleviating this problem by overlapping computation and communication tasks within a single MoE layer, which fails to achieve sufficient overlap and results in limited performance gains. In this work, we introduce an orthogonal partitioning dimension from existing task-parallel methods by leveraging the autoregressive nature of causal Transformer-based LLMs, i.e. partitioning tasks along the sequence dimension. This provides more flexible and efficient overlaps among tasks from both non-MoE and MoE layers. To this end, we propose an efficient MoE training approach, SP-MoE, with two innovative designs. 1) It incorporates non-MoE layers into the overlapping with not only the current MoE layer but also the preceding MoE layer, thereby facilitating more efficient training; 2) It identifies the optimal combination of pipeline degrees for non-MoE and MoE layers and devises the best scheduling plans for load-imbalanced non-MoE and uniform MoE layers to achieve the goal of minimizing the total training latency. Furthermore, we extend our algorithm to incorporate pipeline parallelism (PP) for training larger MoE models, which introduces peer-to-peer (P2P) communications between adjacent PP stages. To mitigate this problem, we propose S3P-MoE with two optimizations: 1) it partitions P2P communication into chunks and each P2P chunk is transmitted immediately to be overlapped with expert computation; 2) it allows the subsequent PP stage to commence upon receiving the corresponding P2P chunk. Moreover, we refine our algorithm to jointly optimize pipeline degrees for both MoE and non-MoE layers, as well as P2P communication, to minimize overall training latency. Extensive experiments demonstrate that SP-MoE and S3P-MoE effectively identify optimal pipeline degrees, reducing training time by up to $34.3\%$ 34.3 % and $30.61\%$ 30.61 % , respectively, compared to state-of-the-art methods.

Förderung

  • National Natural Science Foundation of China
  • Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies
  • Shenzhen Science and Technology Program
  • National Key Research and Development Program of China
  • Central University Basic Research Fund of China
  • RGC RIF
  • RGC TRS
  • RGC CRF

Zitieren

APA

Wang, N., Lin, W., Zhang, L., Shi, S., Zhou, R., & Li, B. (2026). Efficient Mixture-of-Experts Training With Pipelining Optimization. IEEE Transactions on Computers, 75(10), 3528-3543. https://doi.org/10.1109/tc.2026.3702749

Vancouver

Wang N, Lin W, Zhang L, Shi S, Zhou R, Li B. Efficient Mixture-of-Experts Training With Pipelining Optimization. IEEE Transactions on Computers. 2026;75(10):3528-3543. doi:10.1109/tc.2026.3702749

BibTeX
@article{Wang2026,
  author = {Wang, Ne and Lin, Wenxiang and Zhang, Lin and Shi, Shaohuai and Zhou, Ruiting and Li, Bo},
  title = {Efficient Mixture-of-Experts Training With Pipelining Optimization},
  journal = {IEEE Transactions on Computers},
  year = {2026},
  volume = {75},
  number = {10},
  pages = {3528--3543},
  doi = {10.1109/tc.2026.3702749}
}
Kostenlos starten →