EUVIMEDEUVIMED entdecken
Crossref

Global and Local Semantic Completion Learning for Vision-Language Pre-Training

Rong-Cheng Tu, Yatai Ji, Jie Jiang, Weijie Kong, Chengfei Cai, Wenzhe Zhao, Hongfa Wang, Yujiu Yang, Wei Liu

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-12 · 47 (12) · 11065-11079

DOI: 10.1109/tpami.2025.3596394 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-12. Verfasst von Rong-Cheng Tu und 8 weiteren Autor:innen. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 63 Literaturangaben. Gefördert von: Shenzhen Science and Technology Program, Tsinghua University - Tencent Joint Laboratory for Internet Innovation Technology, National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Cross-modal alignment plays a crucial role in vision-language pre-training (VLP) models, enabling them to capture meaningful associations across different modalities. For this purpose, inspired by the success of masked language modeling (MLM) tasks in the NLP pre-training area, numerous masked modeling tasks have been proposed for VLP to further promote cross-modal interactions. The core idea of previous masked modeling tasks is to focus on reconstructing the masked tokens based on visible context for learning local-local alignment, i.e., associations between image patches and text tokens. However, most of them pay little attention to the global semantic features generated for the masked data, resulting in a limited cross-modal alignment ability of global representations to local features of the other modality. Therefore, in this paper, we propose a novel Global and Local Semantic Completion Learning (GLSCL) task to facilitate global-local alignment and local-local alignment simultaneously. Specifically, the GLSCL task complements the missing semantics of masked data and recovers global and local features by cross-modal interactions. Our GLSCL consists of masked global semantic completion (MGSC) and masked local token completion (MLTC). MGSC promotes learning more representative global features, which have a great impact on the performance of downstream tasks, while MLTC reconstructs modal-fusion local tokens, further enhancing accurate comprehension of multimodal data. To evaluate the proposed approaches on cross-modal alignment, we develop a validation benchmark called ALIGN-BENCH. Moreover, we present a flexible vision encoder, enabling our model to simultaneously perform image-text and video-text multimodal tasks. Experimental results show that our proposed method obtains state-of-the-art performance on various vision-language benchmarks, such as visual question answering, image-text retrieval, and video-text retrieval.

Förderung

  • Shenzhen Science and Technology Program
  • Tsinghua University - Tencent Joint Laboratory for Internet Innovation Technology
  • National Natural Science Foundation of China

Zitieren

APA

Tu, R. C., Ji, Y., Jiang, J., Kong, W., Cai, C., Zhao, W., Wang, H., Yang, Y., & Liu, W. (2025). Global and Local Semantic Completion Learning for Vision-Language Pre-Training. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(12), 11065-11079. https://doi.org/10.1109/tpami.2025.3596394

Vancouver

Tu RC, Ji Y, Jiang J, Kong W, Cai C, Zhao W, et al. Global and Local Semantic Completion Learning for Vision-Language Pre-Training. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(12):11065-11079. doi:10.1109/tpami.2025.3596394

BibTeX
@article{Tu2025,
  author = {Tu, Rong-Cheng and Ji, Yatai and Jiang, Jie and Kong, Weijie and Cai, Chengfei and Zhao, Wenzhe and Wang, Hongfa and Yang, Yujiu and Liu, Wei},
  title = {Global and Local Semantic Completion Learning for Vision-Language Pre-Training},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2025},
  volume = {47},
  number = {12},
  pages = {11065--11079},
  doi = {10.1109/tpami.2025.3596394}
}
Kostenlos starten →