Composed Image Retrieval using Contrastive Learning and Task-oriented CLIP-based Features
ACM Transactions on Multimedia Computing, Communications, and Applications · 31.03.2024 · 20 (3) · 1-24
DOI: 10.1145/3617597 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 23.10.2023. Verfasst von Alberto Baldrati und 3 weiteren Autor:innen. Die Arbeit wurde bisher 65-mal zitiert (Crossref) und nennt 56 Literaturangaben. Gefördert von: European Commission under European Horizon 2020 Programme.
Abstract (Originalsprache)
Given a query composed of a reference image and a relative caption, the Composed Image Retrieval goal is to retrieve images visually similar to the reference one that integrates the modifications expressed by the caption. Given that recent research has demonstrated the efficacy of large-scale vision and language pre-trained (VLP) models in various tasks, we rely on features from the OpenAI CLIP model to tackle the considered task. We initially perform a task-oriented fine-tuning of both CLIP encoders using the element-wise sum of visual and textual features. Then, in the second stage, we train a Combiner network that learns to combine the image-text features integrating the bimodal information and providing combined features used to perform the retrieval. We use contrastive learning in both stages of training. Starting from the bare CLIP features as a baseline, experimental results show that the task-oriented fine-tuning and the carefully crafted Combiner network are highly effective and outperform more complex state-of-the-art approaches on FashionIQ and CIRR, two popular and challenging datasets for composed image retrieval. Code and pre-trained models are available at https://github.com/ABaldrati/CLIP4Cir .
Förderung
- European Commission under European Horizon 2020 Programme
Zitieren
APA
Baldrati, A., Bertini, M., Uricchio, T., & Bimbo, A. D. (2024). Composed Image Retrieval using Contrastive Learning and Task-oriented CLIP-based Features. ACM Transactions on Multimedia Computing, Communications, and Applications, 20(3), 1-24. https://doi.org/10.1145/3617597
Vancouver
Baldrati A, Bertini M, Uricchio T, Bimbo AD. Composed Image Retrieval using Contrastive Learning and Task-oriented CLIP-based Features. ACM Transactions on Multimedia Computing, Communications, and Applications. 2024;20(3):1-24. doi:10.1145/3617597
BibTeX
@article{Baldrati2024,
author = {Baldrati, Alberto and Bertini, Marco and Uricchio, Tiberio and Bimbo, Alberto Del},
title = {Composed Image Retrieval using Contrastive Learning and Task-oriented CLIP-based Features},
journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
year = {2024},
volume = {20},
number = {3},
pages = {1--24},
doi = {10.1145/3617597}
}
EUVIMED