EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

MIME: Mutual information-guided mixture of experts for robust multimodal sentiment analysis

Dehai Zhang, Qiang Xiao, Xu He, Jiahao Zhang

Journal of King Saud University Computer and Information Sciences · 2026-07 · 38 (5)

DOI: 10.1007/s44443-026-00587-9 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of King Saud University Computer and Information Sciences, online erschienen am 14.03.2026. Verfasst von Dehai Zhang und 3 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 59 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: National Natural Science Foundation of China, Yunnan Fundamental Research Projects.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Multimodal Sentiment Analysis (MSA) aims to predict human emotions by integrating textual, acoustic, and visual signals. While existing methods have achieved remarkable progress, most approaches treat all modalities equally without considering their inherent reliability differences. In real-world scenarios, acoustic and visual modalities are often corrupted by noise or occlusion, whereas textual signals typically remain semantically reliable. We propose MIME (Mutual Information-guided Mixture of Experts), a novel framework that leverages text as a semantic anchor to guide non-verbal modality learning. MIME introduces modality-specific MoE layers for audio and visual modalities, where multiple lightweight experts offer diverse transformation pathways. Unlike conventional routing based solely on attention or similarity, we propose a hybrid routing mechanism combining content-based matching with MI-guided semantic modulation. The MI factor acts as a semantic filter that suppresses experts whose outputs deviate from the textual anchor while amplifying those with high cross-modal alignment. Furthermore, we design an Early-MI objective that maximizes audio-text and visual-text MI for cross-modal alignment while minimizing audio-visual MI to encourage complementary representations. Experiments on CMU-MOSI and CMU-MOSEI demonstrate that MIME achieves competitive performance and exhibits improved robustness under various noise conditions.

Förderung

  • National Natural Science Foundation of China
  • Yunnan Fundamental Research Projects

Zitieren

APA

Zhang, D., Xiao, Q., He, X., & Zhang, J. (2026). MIME: Mutual information-guided mixture of experts for robust multimodal sentiment analysis. Journal of King Saud University Computer and Information Sciences, 38(5). https://doi.org/10.1007/s44443-026-00587-9

Vancouver

Zhang D, Xiao Q, He X, Zhang J. MIME: Mutual information-guided mixture of experts for robust multimodal sentiment analysis. Journal of King Saud University Computer and Information Sciences. 2026;38(5). doi:10.1007/s44443-026-00587-9

BibTeX
@article{Zhang2026,
  author = {Zhang, Dehai and Xiao, Qiang and He, Xu and Zhang, Jiahao},
  title = {MIME: Mutual information-guided mixture of experts for robust multimodal sentiment analysis},
  journal = {Journal of King Saud University Computer and Information Sciences},
  year = {2026},
  volume = {38},
  number = {5},
  doi = {10.1007/s44443-026-00587-9}
}
Kostenlos starten →