EUVIMEDEUVIMED entdecken
Crossref

SoftCLIP: Softer Cross-Modal Alignment Makes CLIP Stronger

Yuting Gao, Jinfeng Liu, Zihan Xu, Tong Wu, Enwei Zhang, Ke Li, Jie Yang, Wei Liu, Xing Sun

Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (3) · 1860-1868

DOI: 10.1609/aaai.v38i3.27955 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Yuting Gao und 8 weiteren Autor:innen. Die Arbeit wurde bisher 53-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

During the preceding biennium, vision-language pre-training has achieved noteworthy success on several downstream tasks. Nevertheless, acquiring high-quality image-text pairs, where the pairs are entirely exclusive of each other, remains a challenging task, and noise exists in the commonly used datasets. To address this issue, we propose SoftCLIP, a novel approach that relaxes the strict one-to-one constraint and achieves a soft cross-modal alignment by introducing a softened target, which is generated from the fine-grained intra-modal self-similarity. The intra-modal guidance is indicative to enable two pairs have some local similarities and model many-to-many relationships between the two modalities. Besides, since the positive still dominates in the softened target distribution, we disentangle the negatives in the distribution to further boost the relation alignment with the negatives in the cross-modal learning. Extensive experiments demonstrate the effectiveness of SoftCLIP. In particular, on ImageNet zero-shot classification task, using CC3M/CC12M as pre-training dataset, SoftCLIP brings a top-1 accuracy improvement of 6.8%/7.2% over the CLIP baseline.

Zitieren

APA

Gao, Y., Liu, J., Xu, Z., Wu, T., Zhang, E., Li, K., Yang, J., Liu, W., & Sun, X. (2024). SoftCLIP: Softer Cross-Modal Alignment Makes CLIP Stronger. Proceedings of the AAAI Conference on Artificial Intelligence, 38(3), 1860-1868. https://doi.org/10.1609/aaai.v38i3.27955

Vancouver

Gao Y, Liu J, Xu Z, Wu T, Zhang E, Li K, et al. SoftCLIP: Softer Cross-Modal Alignment Makes CLIP Stronger. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(3):1860-1868. doi:10.1609/aaai.v38i3.27955

BibTeX
@article{Gao2024,
  author = {Gao, Yuting and Liu, Jinfeng and Xu, Zihan and Wu, Tong and Zhang, Enwei and Li, Ke and Yang, Jie and Liu, Wei and Sun, Xing},
  title = {SoftCLIP: Softer Cross-Modal Alignment Makes CLIP Stronger},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2024},
  volume = {38},
  number = {3},
  pages = {1860--1868},
  doi = {10.1609/aaai.v38i3.27955}
}
Kostenlos starten →