EUVIMEDEUVIMED entdecken
Crossref

SMIDT: High-Performance Inference Framework for MoE Models with Dynamic Top-K Routing

Zewen Jin, Shen Fu, Chengjie Tang, Youhui Bai, Shengnan Wang, Jiaan Zhu, Chizheng Fang, Ping Gong, Cheng Li

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (27) · 22444-22453

DOI: 10.1609/aaai.v40i27.39403 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Zewen Jin und 8 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

To accelerate Mixture-of-Experts (MoE) inference, the hybrid parallelism paradigm is first applying pipeline parallelism (PP) to vertically divide the model into stages, with each stage further divided horizontally using tensor or expert parallelism. On the algorithm side, dynamic Top-K routing reduces computation by activating fewer experts per token on average. In this paper, we explore the application of dynamic Top-K routing to PP-enabled MoE inference, aiming to fully unleash their combined potential. We identify key performance bottlenecks arising from Top-K value variation across layers, which conflicts with PP's typically uniform stage partitioning, as well as opportunities to optimize memory usage through their integration. To address these challenges, we present SMIDT, an efficient MoE inference framework tailored for dynamic Top-K routing. SMIDT features: (1) an adaptive, module-level uneven partitioning strategy to balance computation across PP stages, (2) a memory-aware expert replication scheme (DPMoE) that reduces communication overhead, and (3) a lightweight search algorithm combining binary search and dynamic programming to generate efficient parallelism plans. We implement SMIDT on SGLang, a state-of-the-art LLM inference framework, evaluate it on 32 A40 GPUs and 16 A100 GPUs, and compare with manually tuned parallelism strategies. Experimental results show that, when co-locating prefill and decoding phases, SMIDT achieves 1.20–3.13x throughput improvements for prefill-only tasks and 1.05–1.89x for prefill-decoding tasks. When disaggregating prefill and decoding tasks, SMIDT improves average and P99 time-to-first-token (TTFT) by 1.10–1.17x and 1.21–1.26x, respectively.

Zitieren

APA

Jin, Z., Fu, S., Tang, C., Bai, Y., Wang, S., Zhu, J., Fang, C., Gong, P., & Li, C. (2026). SMIDT: High-Performance Inference Framework for MoE Models with Dynamic Top-K Routing. Proceedings of the AAAI Conference on Artificial Intelligence, 40(27), 22444-22453. https://doi.org/10.1609/aaai.v40i27.39403

Vancouver

Jin Z, Fu S, Tang C, Bai Y, Wang S, Zhu J, et al. SMIDT: High-Performance Inference Framework for MoE Models with Dynamic Top-K Routing. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(27):22444-22453. doi:10.1609/aaai.v40i27.39403

BibTeX
@article{Jin2026,
  author = {Jin, Zewen and Fu, Shen and Tang, Chengjie and Bai, Youhui and Wang, Shengnan and Zhu, Jiaan and Fang, Chizheng and Gong, Ping and Li, Cheng},
  title = {SMIDT: High-Performance Inference Framework for MoE Models with Dynamic Top-K Routing},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {27},
  pages = {22444--22453},
  doi = {10.1609/aaai.v40i27.39403}
}
Kostenlos starten →