To Align or Not to Align: Strategic Multimodal Representation Alignment for Optimal Performance
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (25) · 21056-21064
DOI: 10.1609/aaai.v40i25.39248 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Wanlong Fang und 2 weiteren Autor:innen. Die Arbeit wurde bisher 9-mal zitiert (Crossref).
Abstract (Originalsprache)
Multimodal learning often relies on aligning representations across modalities to enable effective information integration—an approach traditionally assumed to be universally beneficial. However, prior research has primarily taken an observational approach, examining naturally occurring alignment in multimodal data and exploring its correlation with model performance, without systematically studying the direct effects of explicitly enforced alignment between representations of different modalities. In this work, we investigate how explicit alignment influences both model performance and representation alignment under different modality-specific information structures. Specifically, we introduce a controllable contrastive learning module that enables precise manipulation of alignment strength during training, allowing us to explore when explicit alignment improves or hinders performance. Our results on synthetic and real datasets under different data characteristics show that the impact of explicit alignment on the performance of unimodal models is related to the characteristics of the data: the optimal level of alignment depends on the amount of redundancy between the different modalities. We can find an optimal alignment strength that balances modality-specific signals and shared redundancy in the mixed information distributions. This work can help practitioners on when and how to enforce alignment for optimal unimodal encoder performance.
Zitieren
APA
Fang, W., Zhang, T., & Chan, A. (2026). To Align or Not to Align: Strategic Multimodal Representation Alignment for Optimal Performance. Proceedings of the AAAI Conference on Artificial Intelligence, 40(25), 21056-21064. https://doi.org/10.1609/aaai.v40i25.39248
Vancouver
Fang W, Zhang T, Chan A. To Align or Not to Align: Strategic Multimodal Representation Alignment for Optimal Performance. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(25):21056-21064. doi:10.1609/aaai.v40i25.39248
BibTeX
@article{Fang2026,
author = {Fang, Wanlong and Zhang, Tianle and Chan, Alvin},
title = {To Align or Not to Align: Strategic Multimodal Representation Alignment for Optimal Performance},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {25},
pages = {21056--21064},
doi = {10.1609/aaai.v40i25.39248}
}
EUVIMED