EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

MissModal: Increasing Robustness to Missing Modality in Multimodal Sentiment Analysis

Ronghao Lin, Haifeng Hu

Transactions of the Association for Computational Linguistics · 19.12.2023 · 11 · 1686-1702

DOI: 10.1162/tacl_a_00628 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Transactions of the Association for Computational Linguistics, online erschienen am 19.12.2023. Verfasst von Ronghao Lin und 1 weiteren Autor:innen. Die Arbeit wurde bisher 87-mal zitiert (Crossref) und nennt 40 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

When applying multimodal machine learning in downstream inference, both joint and coordinated multimodal representations rely on the complete presence of modalities as in training. However, modal-incomplete data, where certain modalities are missing, greatly reduces performance in Multimodal Sentiment Analysis (MSA) due to varying input forms and semantic information deficiencies. This limits the applicability of the predominant MSA methods in the real world, where the completeness of multimodal data is uncertain and variable. The generation-based methods attempt to generate the missing modality, yet they require complex hierarchical architecture with huge computational costs and struggle with the representation gaps across different modalities. Diversely, we propose a novel representation learning approach named MissModal, devoting to increasing robustness to missing modality in a classification approach. Specifically, we adopt constraints with geometric contrastive loss, distribution distance loss, and sentiment semantic loss to align the representations of modal-missing and modal-complete data, without impacting the sentiment inference for the complete modalities. Furthermore, we do not demand any changes in the multimodal fusion stage, highlighting the generality of our method in other multimodal learning systems. Extensive experiments demonstrate that the proposed method achieves superior performance with minimal computational costs in various missing modalities scenarios (flexibility), including severely missing modality (efficiency) on two public MSA datasets.

Zitieren

APA

Lin, R., & Hu, H. (2023). MissModal: Increasing Robustness to Missing Modality in Multimodal Sentiment Analysis. Transactions of the Association for Computational Linguistics, 11, 1686-1702. https://doi.org/10.1162/tacl_a_00628

Vancouver

Lin R, Hu H. MissModal: Increasing Robustness to Missing Modality in Multimodal Sentiment Analysis. Transactions of the Association for Computational Linguistics. 2023;11:1686-1702. doi:10.1162/tacl_a_00628

BibTeX
@article{Lin2023,
  author = {Lin, Ronghao and Hu, Haifeng},
  title = {MissModal: Increasing Robustness to Missing Modality in Multimodal Sentiment Analysis},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2023},
  volume = {11},
  pages = {1686--1702},
  doi = {10.1162/tacl_a_00628}
}
Kostenlos starten →