Image–Text Matching Model Based on CLIP Bimodal Encoding
Applied Sciences · 12.11.2024 · 14 (22) · 10384
DOI: 10.3390/app142210384 ↗Kurz gefasst
Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 12.11.2024. Verfasst von Yihuan Zhu und 3 weiteren Autor:innen. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 24 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Jilin Provincial Department of Education.
Abstract (Originalsprache)
Image–text matching is a fundamental task in the multimodal research field, connecting computer vision and natural language processing by aligning visual content with corresponding textual descriptions. Accurate matching is critical for applications such as image captioning and text-based image retrieval yet remains challenging due to the differences in data modalities. This paper addresses these challenges by proposing a robust image–text matching model inspired by Contrastive Language–Image Pre-training (CLIP). Our approach employs the Vision Transformer (ViT) model as the image encoder and Bidirectional Encoder Representations from Transformers (Bert) as the text encoder, integrating these into a shared vector space to measure semantic similarity. We enhance the model’s training efficiency using the LiT-tuning paradigm to optimize learning through a cosine decay strategy for dynamic adjustment of the learning rate. We validate our method on two benchmark datasets, WuKong and Flickr30k, demonstrating that our model achieves superior performance and significantly improves key evaluation metrics. The results underscore the model’s effectiveness in achieving accurate and robust image–text alignment.
Förderung
- Jilin Provincial Department of Education
Zitieren
APA
Zhu, Y., Xu, H., Du, A., & Wang, B. (2024). Image–Text Matching Model Based on CLIP Bimodal Encoding. Applied Sciences, 14(22), 10384. https://doi.org/10.3390/app142210384
Vancouver
Zhu Y, Xu H, Du A, Wang B. Image–Text Matching Model Based on CLIP Bimodal Encoding. Applied Sciences. 2024;14(22):10384. doi:10.3390/app142210384
BibTeX
@article{Zhu2024,
author = {Zhu, Yihuan and Xu, Honghua and Du, Ailin and Wang, Bin},
title = {Image–Text Matching Model Based on CLIP Bimodal Encoding},
journal = {Applied Sciences},
year = {2024},
volume = {14},
number = {22},
pages = {10384},
doi = {10.3390/app142210384}
}
EUVIMED