Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026-03 · 48 (3) · 2329-2341
DOI: 10.1109/tpami.2025.3629605 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026-03. Verfasst von Li Shen und 8 weiteren Autor:innen. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 84 Literaturangaben. Gefördert von: NTURSR, StartUpGrants, National Natural Science Foundation of China.
Abstract (Originalsprache)
Multi-task learning (MTL) leverages a shared model to accomplish multiple tasks and facilitate knowledge transfer. Recent research on task arithmetic-based MTL demonstrates that merging the parameters of independently fine-tuned models can effectively achieve MTL. However, existing merging methods primarily seek a static optimal solution within the original model parameter space, which often results in performance degradation due to the inherent diversity among tasks and potential interferences. To address this challenge, in this paper, we propose a Weight-Ensembling Mixture of Experts (WEMoE) method for multi-task model merging. Specifically, we first identify critical (or sensitive) modules by analyzing parameter variations in core modules of Transformer-based models before and after fine-tuning. Then, our WEMoE statically merges non-critical modules while transforming critical modules into a mixture-of-experts (MoE) structure. During inference, expert modules in the MoE are dynamically merged based on input samples, enabling a more flexible and adaptive merging approach. Building on WEMoE, we further introduce an efficient-and-effective WEMoE (E-WEMoE) method, whose core mechanism involves eliminating non-essential elements in the critical modules of WEMoE and implementing shared routing across multiple MoE modules, thereby significantly reducing both the trainable parameters, the overall parameter count, and computational overhead of the merged model by WEMoE. Experimental results across various architectures and tasks demonstrate that both WEMoE and E-WEMoE outperform state-of-the-art (SOTA) model merging methods in terms of MTL performance, generalization, and robustness.
Förderung
- NTURSR
- StartUpGrants
- National Natural Science Foundation of China
- Shenzhen Municipal Fundamental Research Program
- Natural Science Foundation
- Basic Research Key Project
- Foundation for Innovative Research Groups of Hubei Province
- Science and Technology Major Project of Hubei Province
Zitieren
APA
Shen, L., Tang, A., Yang, E., Guo, G., Luo, Y., Zhang, L., Cao, X., Du, B., & Tao, D. (2026). Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(3), 2329-2341. https://doi.org/10.1109/tpami.2025.3629605
Vancouver
Shen L, Tang A, Yang E, Guo G, Luo Y, Zhang L, et al. Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2026;48(3):2329-2341. doi:10.1109/tpami.2025.3629605
BibTeX
@article{Shen2026,
author = {Shen, Li and Tang, Anke and Yang, Enneng and Guo, Guibing and Luo, Yong and Zhang, Lefei and Cao, Xiaochun and Du, Bo and Tao, Dacheng},
title = {Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2026},
volume = {48},
number = {3},
pages = {2329--2341},
doi = {10.1109/tpami.2025.3629605}
}
EUVIMED