EUVIMEDEUVIMED entdecken
Crossref

Robust Multimodal Representation under Uncertain Missing Modalities

Guilin Lan, Yeqian Du, Zhouwang Yang

ACM Transactions on Multimedia Computing, Communications, and Applications · 31.01.2025 · 21 (1) · 1-23

DOI: 10.1145/3702003 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 21.12.2024. Verfasst von Guilin Lan und 2 weiteren Autor:innen. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 58 Literaturangaben. Gefördert von: National Key R&D Program of China, NSF of China, Major Project of Science and Technology Innovation Tackling Plan of Anhui Province.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Multimodal representation learning has gained significant attention across various fields, yet it faces challenges when dealing with missing modalities in real-world applications. Existing solutions are confined to specific scenarios, such as single-modality missing or missing modalities in test cases, thereby restricting their applicability. To address a more general scenario of uncertain missing modalities in both training and testing phases, we propose Robust Multimodal Representation under Uncertain Missing Modalities (RMRU). This framework projects each modality’s representation into a shared subspace, enabling the reconstruction of any missing modalities within a unified model. We propose an interaction refinement module that utilizes cross-modal attention to enhance these reconstructions, particularly beneficial in scenarios with limited complete modality data. Furthermore, we introduce an iterative training strategy that alternately trains different modules to effectively utilize both complete and incomplete modality data. Experimental results on four benchmark datasets demonstrate the superiority of RMRU over existing baselines, particularly in scenarios with a high rate of missing modalities. Remarkably, our proposed RMRU can be broadly applied to diverse scenarios, regardless of modality types and quantities.

Förderung

  • National Key R&D Program of China
  • NSF of China
  • Major Project of Science and Technology Innovation Tackling Plan of Anhui Province

Zitieren

APA

Lan, G., Du, Y., & Yang, Z. (2025). Robust Multimodal Representation under Uncertain Missing Modalities. ACM Transactions on Multimedia Computing, Communications, and Applications, 21(1), 1-23. https://doi.org/10.1145/3702003

Vancouver

Lan G, Du Y, Yang Z. Robust Multimodal Representation under Uncertain Missing Modalities. ACM Transactions on Multimedia Computing, Communications, and Applications. 2025;21(1):1-23. doi:10.1145/3702003

BibTeX
@article{Lan2025,
  author = {Lan, Guilin and Du, Yeqian and Yang, Zhouwang},
  title = {Robust Multimodal Representation under Uncertain Missing Modalities},
  journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
  year = {2025},
  volume = {21},
  number = {1},
  pages = {1--23},
  doi = {10.1145/3702003}
}
Kostenlos starten →