Global and Local Semantic Completion Learning for Vision-Language Pre-Training
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-12 · 47 (12) · 11065-11079
DOI: 10.1109/tpami.2025.3596394 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-12. Verfasst von Rong-Cheng Tu und 8 weiteren Autor:innen. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 63 Literaturangaben. Gefördert von: Shenzhen Science and Technology Program, Tsinghua University - Tencent Joint Laboratory for Internet Innovation Technology, National Natural Science Foundation of China.
Abstract (Originalsprache)
Cross-modal alignment plays a crucial role in vision-language pre-training (VLP) models, enabling them to capture meaningful associations across different modalities. For this purpose, inspired by the success of masked language modeling (MLM) tasks in the NLP pre-training area, numerous masked modeling tasks have been proposed for VLP to further promote cross-modal interactions. The core idea of previous masked modeling tasks is to focus on reconstructing the masked tokens based on visible context for learning local-local alignment, i.e., associations between image patches and text tokens. However, most of them pay little attention to the global semantic features generated for the masked data, resulting in a limited cross-modal alignment ability of global representations to local features of the other modality. Therefore, in this paper, we propose a novel Global and Local Semantic Completion Learning (GLSCL) task to facilitate global-local alignment and local-local alignment simultaneously. Specifically, the GLSCL task complements the missing semantics of masked data and recovers global and local features by cross-modal interactions. Our GLSCL consists of masked global semantic completion (MGSC) and masked local token completion (MLTC). MGSC promotes learning more representative global features, which have a great impact on the performance of downstream tasks, while MLTC reconstructs modal-fusion local tokens, further enhancing accurate comprehension of multimodal data. To evaluate the proposed approaches on cross-modal alignment, we develop a validation benchmark called ALIGN-BENCH. Moreover, we present a flexible vision encoder, enabling our model to simultaneously perform image-text and video-text multimodal tasks. Experimental results show that our proposed method obtains state-of-the-art performance on various vision-language benchmarks, such as visual question answering, image-text retrieval, and video-text retrieval.
Förderung
- Shenzhen Science and Technology Program
- Tsinghua University - Tencent Joint Laboratory for Internet Innovation Technology
- National Natural Science Foundation of China
Zitieren
APA
Tu, R. C., Ji, Y., Jiang, J., Kong, W., Cai, C., Zhao, W., Wang, H., Yang, Y., & Liu, W. (2025). Global and Local Semantic Completion Learning for Vision-Language Pre-Training. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(12), 11065-11079. https://doi.org/10.1109/tpami.2025.3596394
Vancouver
Tu RC, Ji Y, Jiang J, Kong W, Cai C, Zhao W, et al. Global and Local Semantic Completion Learning for Vision-Language Pre-Training. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(12):11065-11079. doi:10.1109/tpami.2025.3596394
BibTeX
@article{Tu2025,
author = {Tu, Rong-Cheng and Ji, Yatai and Jiang, Jie and Kong, Weijie and Cai, Chengfei and Zhao, Wenzhe and Wang, Hongfa and Yang, Yujiu and Liu, Wei},
title = {Global and Local Semantic Completion Learning for Vision-Language Pre-Training},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2025},
volume = {47},
number = {12},
pages = {11065--11079},
doi = {10.1109/tpami.2025.3596394}
}
EUVIMED