EUVIMEDEUVIMED entdecken
Crossref

Fine-Grained Alignment Network for Zero-Shot Cross-Modal Retrieval

Shiping Ge, Zhiwei Jiang, Yafeng Yin, Cong Wang, Zifeng Cheng, Qing Gu

ACM Transactions on Multimedia Computing, Communications, and Applications · 31.10.2025 · 21 (10) · 1-24

DOI: 10.1145/3722223 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 14.10.2025. Verfasst von Shiping Ge und 5 weiteren Autor:innen. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 65 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Collaborative Innovation Center of Novel Software Technology and Industrialization, and Fundamental Research Funds for the Central Universities.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Zero-Shot Cross-Modal Retrieval (ZS-CMR) aims to perform cross-modal retrieval on data of unseen classes, where a key challenge is how to address the modality-gap and domain-shift problems simultaneously. Existing methods tackle this challenge mainly by embracing a sample-label alignment paradigm, which aligns samples of different modalities but of the same class with the word embedding of their class label. However, these methods only focus on the class-level alignment and overlook the alignment of rich fine-grained semantic information in samples, incurring coarse understanding of sample matching and poor generalization on unseen classes. In this article, we propose a novel Fine-Grained Alignment Network, an end-to-end framework that learns representation with two fine-grained alignment strategies, yielding representation space that can be better generalized to unseen classes. Specifically, we extract two kinds of fine-grained representations, region embedding and label distribution, respectively, from aspects of both feature and label. To optimize the region embedding, we propose a Fine-Grained Contrastive Learning (FGCL) strategy to simultaneously conduct class-level alignment and model the intra-class discrepancy. To optimize the label distribution, we propose a Fine-Grained Label Alignment (FGLA) strategy to align diverse fine-grained semantic information among samples, rather than merely label information. Finally, both region embedding and label distribution are utilized together to perform ZS-CMR at a finer granularity. Experimental results on three widely used datasets demonstrate that our method outperforms the state-of-the-art methods by a large margin. Detailed ablation studies have also been carried out, which provably affirm the advantage of each component we propose. Our code will be available at https://github.com/ShipingGe/FGAN .

Förderung

  • National Natural Science Foundation of China
  • Collaborative Innovation Center of Novel Software Technology and Industrialization, and Fundamental Research Funds for the Central Universities

Zitieren

APA

Ge, S., Jiang, Z., Yin, Y., Wang, C., Cheng, Z., & Gu, Q. (2025). Fine-Grained Alignment Network for Zero-Shot Cross-Modal Retrieval. ACM Transactions on Multimedia Computing, Communications, and Applications, 21(10), 1-24. https://doi.org/10.1145/3722223

Vancouver

Ge S, Jiang Z, Yin Y, Wang C, Cheng Z, Gu Q. Fine-Grained Alignment Network for Zero-Shot Cross-Modal Retrieval. ACM Transactions on Multimedia Computing, Communications, and Applications. 2025;21(10):1-24. doi:10.1145/3722223

BibTeX
@article{Ge2025,
  author = {Ge, Shiping and Jiang, Zhiwei and Yin, Yafeng and Wang, Cong and Cheng, Zifeng and Gu, Qing},
  title = {Fine-Grained Alignment Network for Zero-Shot Cross-Modal Retrieval},
  journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
  year = {2025},
  volume = {21},
  number = {10},
  pages = {1--24},
  doi = {10.1145/3722223}
}
Kostenlos starten →