i-Code: An Integrative and Composable Multimodal Learning Framework
Proceedings of the AAAI Conference on Artificial Intelligence · 26.06.2023 · 37 (9) · 10880-10890
DOI: 10.1609/aaai.v37i9.26290 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 26.06.2023. Verfasst von Ziyi Yang und 19 weiteren Autor:innen. Die Arbeit wurde bisher 19-mal zitiert (Crossref).
Abstract (Originalsprache)
Human intelligence is multimodal; we integrate visual, linguistic, and acoustic signals to maintain a holistic worldview. Most current pretraining methods, however, are limited to one or two modalities. We present i-Code, a self-supervised pretraining framework where users may flexibly combine the modalities of vision, speech, and language into unified and general-purpose vector representations. In this framework, data from each modality are first given to pretrained single-modality encoders. The encoder outputs are then integrated with a multimodal fusion network, which uses novel merge- and co-attention mechanisms to effectively combine information from the different modalities. The entire system is pretrained end-to-end with new objectives including masked modality unit modeling and cross-modality contrastive learning. Unlike previous research using only video for pretraining, the i-Code framework can dynamically process single, dual, and triple-modality data during training and inference, flexibly projecting different combinations of modalities into a single representation space. Experimental results demonstrate how i-Code can outperform state-of-the-art techniques on five multimodal understanding tasks and single-modality benchmarks, improving by as much as 11% and demonstrating the power of integrative multimodal pretraining.
Zitieren
APA
Yang, Z., Fang, Y., Zhu, C., Pryzant, R., Chen, D., Shi, Y., Xu, Y., Qian, Y., Gao, M., Chen, Y., Lu, L., Xie, Y., Gmyr, R., Codella, N., Kanda, N., Xiao, B., Yuan, L., Yoshioka, T., Zeng, M., & Huang, X. (2023). i-Code: An Integrative and Composable Multimodal Learning Framework. Proceedings of the AAAI Conference on Artificial Intelligence, 37(9), 10880-10890. https://doi.org/10.1609/aaai.v37i9.26290
Vancouver
Yang Z, Fang Y, Zhu C, Pryzant R, Chen D, Shi Y, et al. i-Code: An Integrative and Composable Multimodal Learning Framework. Proceedings of the AAAI Conference on Artificial Intelligence. 2023;37(9):10880-10890. doi:10.1609/aaai.v37i9.26290
BibTeX
@article{Yang2023,
author = {Yang, Ziyi and Fang, Yuwei and Zhu, Chenguang and Pryzant, Reid and Chen, DongDong and Shi, Yu and Xu, Yichong and Qian, Yao and Gao, Mei and Chen, Yi-Ling and Lu, Liyang and Xie, Yujia and Gmyr, Robert and Codella, Noel and Kanda, Naoyuki and Xiao, Bin and Yuan, Lu and Yoshioka, Takuya and Zeng, Michael and Huang, Xuedong},
title = {i-Code: An Integrative and Composable Multimodal Learning Framework},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2023},
volume = {37},
number = {9},
pages = {10880--10890},
doi = {10.1609/aaai.v37i9.26290}
}
EUVIMED