Robust Multimodal Representation under Uncertain Missing Modalities
ACM Transactions on Multimedia Computing, Communications, and Applications · 2025-01-31 · 21 (1) · 1-23
DOI: 10.1145/3702003 ↗At a glance
Journal article in ACM Transactions on Multimedia Computing, Communications, and Applications, published online on 2024-12-21. Written by Guilin Lan and 2 co-authors. Cited 13 times so far (Crossref); 58 references. Funded by: National Key R&D Program of China, NSF of China, Major Project of Science and Technology Innovation Tackling Plan of Anhui Province.
Abstract (original language)
Multimodal representation learning has gained significant attention across various fields, yet it faces challenges when dealing with missing modalities in real-world applications. Existing solutions are confined to specific scenarios, such as single-modality missing or missing modalities in test cases, thereby restricting their applicability. To address a more general scenario of uncertain missing modalities in both training and testing phases, we propose Robust Multimodal Representation under Uncertain Missing Modalities (RMRU). This framework projects each modality’s representation into a shared subspace, enabling the reconstruction of any missing modalities within a unified model. We propose an interaction refinement module that utilizes cross-modal attention to enhance these reconstructions, particularly beneficial in scenarios with limited complete modality data. Furthermore, we introduce an iterative training strategy that alternately trains different modules to effectively utilize both complete and incomplete modality data. Experimental results on four benchmark datasets demonstrate the superiority of RMRU over existing baselines, particularly in scenarios with a high rate of missing modalities. Remarkably, our proposed RMRU can be broadly applied to diverse scenarios, regardless of modality types and quantities.
Funding
- National Key R&D Program of China
- NSF of China
- Major Project of Science and Technology Innovation Tackling Plan of Anhui Province
Cite
APA
Lan, G., Du, Y., & Yang, Z. (2025). Robust Multimodal Representation under Uncertain Missing Modalities. ACM Transactions on Multimedia Computing, Communications, and Applications, 21(1), 1-23. https://doi.org/10.1145/3702003
Vancouver
Lan G, Du Y, Yang Z. Robust Multimodal Representation under Uncertain Missing Modalities. ACM Transactions on Multimedia Computing, Communications, and Applications. 2025;21(1):1-23. doi:10.1145/3702003
BibTeX
@article{Lan2025,
author = {Lan, Guilin and Du, Yeqian and Yang, Zhouwang},
title = {Robust Multimodal Representation under Uncertain Missing Modalities},
journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
year = {2025},
volume = {21},
number = {1},
pages = {1--23},
doi = {10.1145/3702003}
}
EUVIMED