EUVIMEDEUVIMED entdecken
Crossref

Uni-MoE: Scaling Unified Multimodal LLMs With Mixture of Experts

Yunxin Li, Shenyuan Jiang, Baotian Hu, Longyue Wang, Wanqi Zhong, Wenhan Luo, Lin Ma, Min Zhang

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-05 · 47 (5) · 3424-3439

DOI: 10.1109/tpami.2025.3532688 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-05. Verfasst von Yunxin Li und 7 weiteren Autor:innen. Die Arbeit wurde bisher 74-mal zitiert (Crossref) und nennt 79 Literaturangaben. Gefördert von: National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recent advancements in Multimodal Large Language Models (MLLMs) underscore the significance of scalable models and data to boost performance, yet this often incurs substantial computational costs. Although the Mixture of Experts (MoE) architecture has been employed to scale large language or visual-language models efficiently, these efforts typically involve fewer experts and limited modalities. To address this, our work presents the pioneering attempt to develop a unified MLLM with the MoE architecture, named Uni-MoE that can handle a wide array of modalities. Specifically, it features modality-specific encoders with connectors for a unified multimodal representation. We also implement a sparse MoE architecture within the LLMs to enable efficient training and inference through modality-level data parallelism and expert-level model parallelism. To enhance the multi-expert collaboration and generalization, we present a progressive training strategy: 1) Cross-modality alignment using various connectors with different cross-modality data, 2) Training modality-specific experts with cross-modality instruction data to activate experts' preferences, and 3) Tuning the whole Uni-MoE framework utilizing Low-Rank Adaptation (LoRA) on mixed multimodal instruction data. We evaluate the instruction-tuned Uni-MoE on a comprehensive set of multimodal datasets. The extensive experimental results demonstrate Uni-MoE's principal advantage of significantly reducing performance bias in handling mixed multimodal datasets, alongside improved multi-expert collaboration and generalization.

Förderung

  • National Natural Science Foundation of China

Zitieren

APA

Li, Y., Jiang, S., Hu, B., Wang, L., Zhong, W., Luo, W., Ma, L., & Zhang, M. (2025). Uni-MoE: Scaling Unified Multimodal LLMs With Mixture of Experts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(5), 3424-3439. https://doi.org/10.1109/tpami.2025.3532688

Vancouver

Li Y, Jiang S, Hu B, Wang L, Zhong W, Luo W, et al. Uni-MoE: Scaling Unified Multimodal LLMs With Mixture of Experts. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(5):3424-3439. doi:10.1109/tpami.2025.3532688

BibTeX
@article{Li2025,
  author = {Li, Yunxin and Jiang, Shenyuan and Hu, Baotian and Wang, Longyue and Zhong, Wanqi and Luo, Wenhan and Ma, Lin and Zhang, Min},
  title = {Uni-MoE: Scaling Unified Multimodal LLMs With Mixture of Experts},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2025},
  volume = {47},
  number = {5},
  pages = {3424--3439},
  doi = {10.1109/tpami.2025.3532688}
}
Kostenlos starten →