EUVIMEDEUVIMED entdecken
Crossref

Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging

Lujun Li, Qiyuan Zhu, Jiacheng Wang, Xiaoyu Qin, Wei Li, Hao Gu, Sirui Han, Yike Guo

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (27) · 22994-23002

DOI: 10.1609/aaai.v40i27.39464 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Lujun Li und 7 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Mixture of Experts (MoE) LLMs face significant obstacles due to their massive parameter scale, which imposes memory, storage, and deployment challenges. Although recent expert merging methods aim to achieve greater efficiency by consolidating several experts, they are fundamentally hindered by parameter conflicts arising from expert specialization. In this paper, we present Sub-MoE, a novel MoE compression framework via Subspace Expert Merging. Our key insight is to perform joint Singular Value Decomposition (SVD) on concatenated expert weights, reducing conflicting parameters by extracting shared U-matrices while enabling effective merging of the expert-specific V components. Specifically, Sub-MoE consists of two innovative stages: (1) Adaptive Expert Clustering, which groups functionally coherent experts via K-means clustering based on cosine similarity of expert outputs; and (2) Subspace Expert Merging, which first performs Experts Union Decomposition to derive the shared U-matrix across experts in the same group, then applies frequency-based merging for individual V-matrices, and completes expert reconstruction using the merged V-matrix. In this way, we align and fuse experts in a shared subspace. Additionally, the framework can be extended with intra-expert compression for further inference optimization. Extensive experiments on Mixtral, DeepSeek, and Qwen-1.5/3 MoE LLMs demonstrate that our Sub-MoE significantly outperforms existing expert pruning and merging methods. Notably, our Sub-MoE maintains 96%/86% of original performance with 25%/50% expert reduction on Mixtral-8×7B in zero-shot benchmarks.

Zitieren

APA

Li, L., Zhu, Q., Wang, J., Qin, X., Li, W., Gu, H., Han, S., & Guo, Y. (2026). Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging. Proceedings of the AAAI Conference on Artificial Intelligence, 40(27), 22994-23002. https://doi.org/10.1609/aaai.v40i27.39464

Vancouver

Li L, Zhu Q, Wang J, Qin X, Li W, Gu H, et al. Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(27):22994-23002. doi:10.1609/aaai.v40i27.39464

BibTeX
@article{Li2026,
  author = {Li, Lujun and Zhu, Qiyuan and Wang, Jiacheng and Qin, Xiaoyu and Li, Wei and Gu, Hao and Han, Sirui and Guo, Yike},
  title = {Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {27},
  pages = {22994--23002},
  doi = {10.1609/aaai.v40i27.39464}
}
Kostenlos starten →