Self-Training Boosted Multi-Factor Matching Network for Composed Image Retrieval
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-05 · 46 (5) · 3665-3678
DOI: 10.1109/tpami.2023.3346434 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-05. Verfasst von Haokun Wen und 5 weiteren Autor:innen. Die Arbeit wurde bisher 31-mal zitiert (Crossref) und nennt 59 Literaturangaben. Gefördert von: China Scholarship Council, National University of Singapore, National Natural Science Foundation of China.
Abstract (Originalsprache)
The composed image retrieval (CIR) task aims to retrieve the desired target image for a given multimodal query, i.e., a reference image with its corresponding modification text. The key limitations encountered by existing efforts are two aspects: 1) ignoring the multiple query-target matching factors; 2) ignoring the potential unlabeled reference-target image pairs in existing benchmark datasets. To address these two limitations is non-trivial due to the following challenges: 1) how to effectively model the multiple matching factors in a latent way without direct supervision signals; 2) how to fully utilize the potential unlabeled reference-target image pairs to improve the generalization ability of the CIR model. To address these challenges, in this work, we first propose a CLIP-Transformer based muLtI-factor Matching Network (LIMN), which consists of three key modules: disentanglement-based latent factor tokens mining, dual aggregation-based matching token learning, and dual query-target matching modeling. Thereafter, we design an iterative dual self-training paradigm to further enhance the performance of LIMN by fully utilizing the potential unlabeled reference-target image pairs in a weakly-supervised manner. Specifically, we denote the iterative dual self-training paradigm enhanced LIMN as LIMN+. Extensive experiments on four datasets, including FashionIQ, Shoes, CIRR, and Fashion200 K, show that our proposed LIMN and LIMN+ significantly surpass the state-of-the-art baselines.
Förderung
- China Scholarship Council
- National University of Singapore
- National Natural Science Foundation of China
- Natural Science Foundation of Shandong Province
- Young Elite Scientists Sponsorship Program by CAST
Zitieren
APA
Wen, H., Song, X., Yin, J., Wu, J., Guan, W., & Nie, L. (2024). Self-Training Boosted Multi-Factor Matching Network for Composed Image Retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(5), 3665-3678. https://doi.org/10.1109/tpami.2023.3346434
Vancouver
Wen H, Song X, Yin J, Wu J, Guan W, Nie L. Self-Training Boosted Multi-Factor Matching Network for Composed Image Retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(5):3665-3678. doi:10.1109/tpami.2023.3346434
BibTeX
@article{Wen2024,
author = {Wen, Haokun and Song, Xuemeng and Yin, Jianhua and Wu, Jianlong and Guan, Weili and Nie, Liqiang},
title = {Self-Training Boosted Multi-Factor Matching Network for Composed Image Retrieval},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2024},
volume = {46},
number = {5},
pages = {3665--3678},
doi = {10.1109/tpami.2023.3346434}
}
EUVIMED