EUVIMEDEUVIMED entdecken
Crossref

Sample-specific Modality Diagnosis and Cross-modal Enhancement for Incomplete Multimodal Representations

Junsong Chen, Jiyuan Liu, Suyuan Liu, Wei Zhang, Ao Li, En Zhu, Xinwang Liu

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (24) · 20154-20162

DOI: 10.1609/aaai.v40i24.39102 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Junsong Chen und 6 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In multimodal sentiment analysis, modality missingness and quality degradation are common. Existing methods often rely on batch-level modality generation, generation but neglect sample-level missingness, hence their flexibility is limited severely in real-world scenarios. To address this, Sample-specific Modality Diagnosis and Cross-modal Enhancement for Incomplete Multimodal Representations (SMCIR) is proposed. Specifically, The Dynamic Multi-feature Fusion Detector (DMFD) is presented, which detects missingness and severity at the sample-level using indicators such as information entropy, modality similarity, and mutual information. Unlike batch-based methods, the DMFD provides fine-grained detection and adaptive responses, improving sensitivity to modality disturbances. Meanwhile, the Context-aware Modality Completion Generator (CMCG) is developed to restore missing modalities through context-guided reconstruction using multiscale feature fusion and cross-modal attention. In this way, the proposed CMCG method can avoid redundancy and inconsistency, enhancing the consistency and discriminativity of the fused representation. In CMCG, the text modality serves as a stable guide to improve context consistency. Experiments on the CMU-MOSI and CMU-MOSEI datasets show that SMCIR outperforms existing full-modal and non-recovery-based methods, well validating its efficacy and superiority in multimodal learning.

Zitieren

APA

Chen, J., Liu, J., Liu, S., Zhang, W., Li, A., Zhu, E., & Liu, X. (2026). Sample-specific Modality Diagnosis and Cross-modal Enhancement for Incomplete Multimodal Representations. Proceedings of the AAAI Conference on Artificial Intelligence, 40(24), 20154-20162. https://doi.org/10.1609/aaai.v40i24.39102

Vancouver

Chen J, Liu J, Liu S, Zhang W, Li A, Zhu E, et al. Sample-specific Modality Diagnosis and Cross-modal Enhancement for Incomplete Multimodal Representations. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(24):20154-20162. doi:10.1609/aaai.v40i24.39102

BibTeX
@article{Chen2026,
  author = {Chen, Junsong and Liu, Jiyuan and Liu, Suyuan and Zhang, Wei and Li, Ao and Zhu, En and Liu, Xinwang},
  title = {Sample-specific Modality Diagnosis and Cross-modal Enhancement for Incomplete Multimodal Representations},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {24},
  pages = {20154--20162},
  doi = {10.1609/aaai.v40i24.39102}
}
Kostenlos starten →