EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE
Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (2) · 1110-1119
DOI: 10.1609/aaai.v38i2.27872 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Junyi Chen und 6 weiteren Autor:innen. Die Arbeit wurde bisher 17-mal zitiert (Crossref).
Abstract (Originalsprache)
Building scalable vision-language models to learn from diverse, multimodal data remains an open challenge. In this paper, we introduce an Efficient Vision-languagE foundation model, namely EVE, which is one unified multimodal Transformer pre-trained solely by one unified pre-training task. Specifically, EVE encodes both vision and language within a shared Transformer network integrated with modality-aware sparse Mixture-of-Experts (MoE) modules, which capture modality-specific information by selectively switching to different experts. To unify pre-training tasks of vision and language, EVE performs masked signal modeling on image-text pairs to reconstruct masked signals, i.e., image pixels and text tokens, given visible signals. This simple yet effective pre-training objective accelerates training by 4x compared to the model pre-trained with Image-Text Contrastive and Image-Text Matching losses. Owing to the combination of the unified architecture and pre-training task, EVE is easy to scale up, enabling better downstream performance with fewer resources and faster training speed. Despite its simplicity, EVE achieves state-of-the-art performance on various vision-language downstream tasks, including visual question answering, visual reasoning, and image-text retrieval.
Zitieren
APA
Chen, J., Guo, L., Sun, J., Shao, S., Yuan, Z., Lin, L., & Zhang, D. (2024). EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE. Proceedings of the AAAI Conference on Artificial Intelligence, 38(2), 1110-1119. https://doi.org/10.1609/aaai.v38i2.27872
Vancouver
Chen J, Guo L, Sun J, Shao S, Yuan Z, Lin L, et al. EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(2):1110-1119. doi:10.1609/aaai.v38i2.27872
BibTeX
@article{Chen2024,
author = {Chen, Junyi and Guo, Longteng and Sun, Jia and Shao, Shuai and Yuan, Zehuan and Lin, Liang and Zhang, Dongyu},
title = {EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2024},
volume = {38},
number = {2},
pages = {1110--1119},
doi = {10.1609/aaai.v38i2.27872}
}
EUVIMED