ViMoE: An Empirical Study of Designing Vision Mixture-of-Experts
IEEE Transactions on Image Processing · 2025 · 34 · 7209-7221
DOI: 10.1109/tip.2025.3626887 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Image Processing, 2025. Verfasst von Xumeng Han und 5 weiteren Autor:innen. Die Arbeit wurde bisher 25-mal zitiert (Crossref) und nennt 56 Literaturangaben. Gefördert von: Key Deployment Program of Chinese Academy of Sciences, China, Strategic Priority Research Program of Chinese Academy of Sciences, China.
Abstract (Originalsprache)
Mixture-of-Experts (MoE) models embody the divide-and-conquer concept and are a promising approach for increasing model capacity, demonstrating excellent scalability across multiple domains. In this paper, we integrate the MoE structure into the classic Vision Transformer (ViT), naming it ViMoE, and explore the potential of applying MoE to vision through a comprehensive study on image classification and semantic segmentation. However, we observe that the performance is sensitive to the configuration of MoE layers, making it challenging to obtain optimal results without careful design. The underlying cause is that inappropriate MoE layers lead to unreliable routing and hinder experts from effectively acquiring helpful information. To address this, we introduce a shared expert to learn and capture common knowledge, serving as an effective way to construct a stable ViMoE. Furthermore, we demonstrate how to analyze expert routing behavior, revealing which MoE layers are capable of specializing in handling specific information and which are not. This provides guidance for retaining the critical layers while removing redundancies, thereby advancing ViMoE to be more efficient without sacrificing accuracy. We aspire for this work to offer new insights into the design of vision MoE models and provide valuable empirical guidance for future research.
Förderung
- Key Deployment Program of Chinese Academy of Sciences, China
- Strategic Priority Research Program of Chinese Academy of Sciences, China
Zitieren
APA
Han, X., Wei, L., Dou, Z., Sun, Y., Han, Z., & Tian, Q. (2025). ViMoE: An Empirical Study of Designing Vision Mixture-of-Experts. IEEE Transactions on Image Processing, 34, 7209-7221. https://doi.org/10.1109/tip.2025.3626887
Vancouver
Han X, Wei L, Dou Z, Sun Y, Han Z, Tian Q. ViMoE: An Empirical Study of Designing Vision Mixture-of-Experts. IEEE Transactions on Image Processing. 2025;34:7209-7221. doi:10.1109/tip.2025.3626887
BibTeX
@article{Han2025,
author = {Han, Xumeng and Wei, Longhui and Dou, Zhiyang and Sun, Yingfei and Han, Zhenjun and Tian, Qi},
title = {ViMoE: An Empirical Study of Designing Vision Mixture-of-Experts},
journal = {IEEE Transactions on Image Processing},
year = {2025},
volume = {34},
pages = {7209--7221},
doi = {10.1109/tip.2025.3626887}
}
EUVIMED