EUVIMEDEUVIMED entdecken
Crossref

EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE

Junyi Chen, Longteng Guo, Jia Sun, Shuai Shao, Zehuan Yuan, Liang Lin, Dongyu Zhang

Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (2) · 1110-1119

DOI: 10.1609/aaai.v38i2.27872 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Junyi Chen und 6 weiteren Autor:innen. Die Arbeit wurde bisher 17-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Building scalable vision-language models to learn from diverse, multimodal data remains an open challenge. In this paper, we introduce an Efficient Vision-languagE foundation model, namely EVE, which is one unified multimodal Transformer pre-trained solely by one unified pre-training task. Specifically, EVE encodes both vision and language within a shared Transformer network integrated with modality-aware sparse Mixture-of-Experts (MoE) modules, which capture modality-specific information by selectively switching to different experts. To unify pre-training tasks of vision and language, EVE performs masked signal modeling on image-text pairs to reconstruct masked signals, i.e., image pixels and text tokens, given visible signals. This simple yet effective pre-training objective accelerates training by 4x compared to the model pre-trained with Image-Text Contrastive and Image-Text Matching losses. Owing to the combination of the unified architecture and pre-training task, EVE is easy to scale up, enabling better downstream performance with fewer resources and faster training speed. Despite its simplicity, EVE achieves state-of-the-art performance on various vision-language downstream tasks, including visual question answering, visual reasoning, and image-text retrieval.

Zitieren

APA

Chen, J., Guo, L., Sun, J., Shao, S., Yuan, Z., Lin, L., & Zhang, D. (2024). EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE. Proceedings of the AAAI Conference on Artificial Intelligence, 38(2), 1110-1119. https://doi.org/10.1609/aaai.v38i2.27872

Vancouver

Chen J, Guo L, Sun J, Shao S, Yuan Z, Lin L, et al. EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(2):1110-1119. doi:10.1609/aaai.v38i2.27872

BibTeX
@article{Chen2024,
  author = {Chen, Junyi and Guo, Longteng and Sun, Jia and Shao, Shuai and Yuan, Zehuan and Lin, Liang and Zhang, Dongyu},
  title = {EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2024},
  volume = {38},
  number = {2},
  pages = {1110--1119},
  doi = {10.1609/aaai.v38i2.27872}
}
Kostenlos starten →