EUVIMEDSkopri EUVIMED
Crossref

Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?

Fei Wang, Liang Ding, Jun Rao, Ye Liu, Li Shen, Changxing Ding

ACM Transactions on Multimedia Computing, Communications, and Applications · 2024-12-31 · 20 (12) · 1-22

DOI: 10.1145/3690640 ↗

At a glance

Journal article in ACM Transactions on Multimedia Computing, Communications, and Applications, published online on 2024-11-18. Written by Fei Wang and 5 co-authors. Cited 10 times so far (Crossref); 69 references. Funded by: National Natural Science Foundation of China, Guangdong Basic and Applied Basic Research Foundation, Guangdong Provincial Key Laboratory of Human Digital Twin.

AstrattDettalji biblijografiċiCiteRelated studies

Abstract (original language)

The field of multimedia research has witnessed significant interest in leveraging multimodal pretrained neural network models to perceive and represent the physical world. Among these models, vision-language pretraining (VLP) has emerged as a captivating topic. Currently, the prevalent approach in VLP involves supervising the training process with paired image-text data. However, limited efforts have been dedicated to exploring the extraction of essential linguistic knowledge, such as semantics and syntax, during VLP and understanding its impact on multimodal alignment. In response, our study aims to shed light on the influence of comprehensive linguistic knowledge encompassing semantic expression and syntactic structure on multimodal alignment. To achieve this, we introduce SNARE , a large-scale multimodal alignment probing benchmark designed specifically for the detection of vital linguistic components, including lexical, semantic, and syntax knowledge. SNARE offers four distinct tasks: Semantic Structure, Negation Logic, Attribute Ownership, and Relationship Composition. Leveraging SNARE , we conduct holistic analyses of six advanced VLP models (BLIP, CLIP, Flava, X-VLM, BLIP2, and GPT-4), along with human performance, revealing key characteristics of the VLP model: (i) Insensitivity to complex syntax structures, relying primarily on content words for sentence comprehension. (ii) Limited comprehension of sentence combinations and negations. (iii) Challenges in determining actions or spatial relations within visual information, as well as difficulties in verifying the correctness of ternary relationships. Based on these findings, we propose the following strategies to enhance multimodal alignment in VLP: (1) Utilize a large generative language model as the language backbone in VLP to facilitate the understanding of complex sentences. (2) Establish high-quality datasets that emphasize content words and employ simple syntax, such as short-distance semantic composition, to improve multimodal alignment. (3) Incorporate more fine-grained visual knowledge, such as spatial relationships, into pretraining objectives. 1

Funding

  • National Natural Science Foundation of China
  • Guangdong Basic and Applied Basic Research Foundation
  • Guangdong Provincial Key Laboratory of Human Digital Twin
  • TCL Young Scholars Program, and the 2024 Tencent AI Lab Rhino-Bird Focused Research Program

Cite

APA

Wang, F., Ding, L., Rao, J., Liu, Y., Shen, L., & Ding, C. (2024). Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?. ACM Transactions on Multimedia Computing, Communications, and Applications, 20(12), 1-22. https://doi.org/10.1145/3690640

Vancouver

Wang F, Ding L, Rao J, Liu Y, Shen L, Ding C. Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?. ACM Transactions on Multimedia Computing, Communications, and Applications. 2024;20(12):1-22. doi:10.1145/3690640

BibTeX
@article{Wang2024,
  author = {Wang, Fei and Ding, Liang and Rao, Jun and Liu, Ye and Shen, Li and Ding, Changxing},
  title = {Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?},
  journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
  year = {2024},
  volume = {20},
  number = {12},
  pages = {1--22},
  doi = {10.1145/3690640}
}
Start for free →