EUVIMEDEUVIMED entdecken
Crossref

Self-Training Boosted Multi-Factor Matching Network for Composed Image Retrieval

Haokun Wen, Xuemeng Song, Jianhua Yin, Jianlong Wu, Weili Guan, Liqiang Nie

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-05 · 46 (5) · 3665-3678

DOI: 10.1109/tpami.2023.3346434 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-05. Verfasst von Haokun Wen und 5 weiteren Autor:innen. Die Arbeit wurde bisher 31-mal zitiert (Crossref) und nennt 59 Literaturangaben. Gefördert von: China Scholarship Council, National University of Singapore, National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The composed image retrieval (CIR) task aims to retrieve the desired target image for a given multimodal query, i.e., a reference image with its corresponding modification text. The key limitations encountered by existing efforts are two aspects: 1) ignoring the multiple query-target matching factors; 2) ignoring the potential unlabeled reference-target image pairs in existing benchmark datasets. To address these two limitations is non-trivial due to the following challenges: 1) how to effectively model the multiple matching factors in a latent way without direct supervision signals; 2) how to fully utilize the potential unlabeled reference-target image pairs to improve the generalization ability of the CIR model. To address these challenges, in this work, we first propose a CLIP-Transformer based muLtI-factor Matching Network (LIMN), which consists of three key modules: disentanglement-based latent factor tokens mining, dual aggregation-based matching token learning, and dual query-target matching modeling. Thereafter, we design an iterative dual self-training paradigm to further enhance the performance of LIMN by fully utilizing the potential unlabeled reference-target image pairs in a weakly-supervised manner. Specifically, we denote the iterative dual self-training paradigm enhanced LIMN as LIMN+. Extensive experiments on four datasets, including FashionIQ, Shoes, CIRR, and Fashion200 K, show that our proposed LIMN and LIMN+ significantly surpass the state-of-the-art baselines.

Förderung

  • China Scholarship Council
  • National University of Singapore
  • National Natural Science Foundation of China
  • Natural Science Foundation of Shandong Province
  • Young Elite Scientists Sponsorship Program by CAST

Zitieren

APA

Wen, H., Song, X., Yin, J., Wu, J., Guan, W., & Nie, L. (2024). Self-Training Boosted Multi-Factor Matching Network for Composed Image Retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(5), 3665-3678. https://doi.org/10.1109/tpami.2023.3346434

Vancouver

Wen H, Song X, Yin J, Wu J, Guan W, Nie L. Self-Training Boosted Multi-Factor Matching Network for Composed Image Retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(5):3665-3678. doi:10.1109/tpami.2023.3346434

BibTeX
@article{Wen2024,
  author = {Wen, Haokun and Song, Xuemeng and Yin, Jianhua and Wu, Jianlong and Guan, Weili and Nie, Liqiang},
  title = {Self-Training Boosted Multi-Factor Matching Network for Composed Image Retrieval},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2024},
  volume = {46},
  number = {5},
  pages = {3665--3678},
  doi = {10.1109/tpami.2023.3346434}
}
Kostenlos starten →