Sample-specific Modality Diagnosis and Cross-modal Enhancement for Incomplete Multimodal Representations
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (24) · 20154-20162
DOI: 10.1609/aaai.v40i24.39102 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Junsong Chen und 6 weiteren Autor:innen.
Abstract (Originalsprache)
In multimodal sentiment analysis, modality missingness and quality degradation are common. Existing methods often rely on batch-level modality generation, generation but neglect sample-level missingness, hence their flexibility is limited severely in real-world scenarios. To address this, Sample-specific Modality Diagnosis and Cross-modal Enhancement for Incomplete Multimodal Representations (SMCIR) is proposed. Specifically, The Dynamic Multi-feature Fusion Detector (DMFD) is presented, which detects missingness and severity at the sample-level using indicators such as information entropy, modality similarity, and mutual information. Unlike batch-based methods, the DMFD provides fine-grained detection and adaptive responses, improving sensitivity to modality disturbances. Meanwhile, the Context-aware Modality Completion Generator (CMCG) is developed to restore missing modalities through context-guided reconstruction using multiscale feature fusion and cross-modal attention. In this way, the proposed CMCG method can avoid redundancy and inconsistency, enhancing the consistency and discriminativity of the fused representation. In CMCG, the text modality serves as a stable guide to improve context consistency. Experiments on the CMU-MOSI and CMU-MOSEI datasets show that SMCIR outperforms existing full-modal and non-recovery-based methods, well validating its efficacy and superiority in multimodal learning.
Zitieren
APA
Chen, J., Liu, J., Liu, S., Zhang, W., Li, A., Zhu, E., & Liu, X. (2026). Sample-specific Modality Diagnosis and Cross-modal Enhancement for Incomplete Multimodal Representations. Proceedings of the AAAI Conference on Artificial Intelligence, 40(24), 20154-20162. https://doi.org/10.1609/aaai.v40i24.39102
Vancouver
Chen J, Liu J, Liu S, Zhang W, Li A, Zhu E, et al. Sample-specific Modality Diagnosis and Cross-modal Enhancement for Incomplete Multimodal Representations. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(24):20154-20162. doi:10.1609/aaai.v40i24.39102
BibTeX
@article{Chen2026,
author = {Chen, Junsong and Liu, Jiyuan and Liu, Suyuan and Zhang, Wei and Li, Ao and Zhu, En and Liu, Xinwang},
title = {Sample-specific Modality Diagnosis and Cross-modal Enhancement for Incomplete Multimodal Representations},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {24},
pages = {20154--20162},
doi = {10.1609/aaai.v40i24.39102}
}
EUVIMED