Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (27) · 22994-23002
DOI: 10.1609/aaai.v40i27.39464 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Lujun Li und 7 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref).
Abstract (Originalsprache)
Mixture of Experts (MoE) LLMs face significant obstacles due to their massive parameter scale, which imposes memory, storage, and deployment challenges. Although recent expert merging methods aim to achieve greater efficiency by consolidating several experts, they are fundamentally hindered by parameter conflicts arising from expert specialization. In this paper, we present Sub-MoE, a novel MoE compression framework via Subspace Expert Merging. Our key insight is to perform joint Singular Value Decomposition (SVD) on concatenated expert weights, reducing conflicting parameters by extracting shared U-matrices while enabling effective merging of the expert-specific V components. Specifically, Sub-MoE consists of two innovative stages: (1) Adaptive Expert Clustering, which groups functionally coherent experts via K-means clustering based on cosine similarity of expert outputs; and (2) Subspace Expert Merging, which first performs Experts Union Decomposition to derive the shared U-matrix across experts in the same group, then applies frequency-based merging for individual V-matrices, and completes expert reconstruction using the merged V-matrix. In this way, we align and fuse experts in a shared subspace. Additionally, the framework can be extended with intra-expert compression for further inference optimization. Extensive experiments on Mixtral, DeepSeek, and Qwen-1.5/3 MoE LLMs demonstrate that our Sub-MoE significantly outperforms existing expert pruning and merging methods. Notably, our Sub-MoE maintains 96%/86% of original performance with 25%/50% expert reduction on Mixtral-8×7B in zero-shot benchmarks.
Zitieren
APA
Li, L., Zhu, Q., Wang, J., Qin, X., Li, W., Gu, H., Han, S., & Guo, Y. (2026). Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging. Proceedings of the AAAI Conference on Artificial Intelligence, 40(27), 22994-23002. https://doi.org/10.1609/aaai.v40i27.39464
Vancouver
Li L, Zhu Q, Wang J, Qin X, Li W, Gu H, et al. Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(27):22994-23002. doi:10.1609/aaai.v40i27.39464
BibTeX
@article{Li2026,
author = {Li, Lujun and Zhu, Qiyuan and Wang, Jiacheng and Qin, Xiaoyu and Li, Wei and Gu, Hao and Han, Sirui and Guo, Yike},
title = {Sub-MoE: Efficient Mixture-of-Expert LLMs Compression via Subspace Expert Merging},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {27},
pages = {22994--23002},
doi = {10.1609/aaai.v40i27.39464}
}
EUVIMED