EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Image–Text Matching Model Based on CLIP Bimodal Encoding

Yihuan Zhu, Honghua Xu, Ailin Du, Bin Wang

Applied Sciences · 12.11.2024 · 14 (22) · 10384

DOI: 10.3390/app142210384 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 12.11.2024. Verfasst von Yihuan Zhu und 3 weiteren Autor:innen. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 24 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Jilin Provincial Department of Education.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Image–text matching is a fundamental task in the multimodal research field, connecting computer vision and natural language processing by aligning visual content with corresponding textual descriptions. Accurate matching is critical for applications such as image captioning and text-based image retrieval yet remains challenging due to the differences in data modalities. This paper addresses these challenges by proposing a robust image–text matching model inspired by Contrastive Language–Image Pre-training (CLIP). Our approach employs the Vision Transformer (ViT) model as the image encoder and Bidirectional Encoder Representations from Transformers (Bert) as the text encoder, integrating these into a shared vector space to measure semantic similarity. We enhance the model’s training efficiency using the LiT-tuning paradigm to optimize learning through a cosine decay strategy for dynamic adjustment of the learning rate. We validate our method on two benchmark datasets, WuKong and Flickr30k, demonstrating that our model achieves superior performance and significantly improves key evaluation metrics. The results underscore the model’s effectiveness in achieving accurate and robust image–text alignment.

Förderung

  • Jilin Provincial Department of Education

Zitieren

APA

Zhu, Y., Xu, H., Du, A., & Wang, B. (2024). Image–Text Matching Model Based on CLIP Bimodal Encoding. Applied Sciences, 14(22), 10384. https://doi.org/10.3390/app142210384

Vancouver

Zhu Y, Xu H, Du A, Wang B. Image–Text Matching Model Based on CLIP Bimodal Encoding. Applied Sciences. 2024;14(22):10384. doi:10.3390/app142210384

BibTeX
@article{Zhu2024,
  author = {Zhu, Yihuan and Xu, Honghua and Du, Ailin and Wang, Bin},
  title = {Image–Text Matching Model Based on CLIP Bimodal Encoding},
  journal = {Applied Sciences},
  year = {2024},
  volume = {14},
  number = {22},
  pages = {10384},
  doi = {10.3390/app142210384}
}
Kostenlos starten →