Fine-Grained Alignment Network for Zero-Shot Cross-Modal Retrieval
ACM Transactions on Multimedia Computing, Communications, and Applications · 31.10.2025 · 21 (10) · 1-24
DOI: 10.1145/3722223 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 14.10.2025. Verfasst von Shiping Ge und 5 weiteren Autor:innen. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 65 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Collaborative Innovation Center of Novel Software Technology and Industrialization, and Fundamental Research Funds for the Central Universities.
Abstract (Originalsprache)
Zero-Shot Cross-Modal Retrieval (ZS-CMR) aims to perform cross-modal retrieval on data of unseen classes, where a key challenge is how to address the modality-gap and domain-shift problems simultaneously. Existing methods tackle this challenge mainly by embracing a sample-label alignment paradigm, which aligns samples of different modalities but of the same class with the word embedding of their class label. However, these methods only focus on the class-level alignment and overlook the alignment of rich fine-grained semantic information in samples, incurring coarse understanding of sample matching and poor generalization on unseen classes. In this article, we propose a novel Fine-Grained Alignment Network, an end-to-end framework that learns representation with two fine-grained alignment strategies, yielding representation space that can be better generalized to unseen classes. Specifically, we extract two kinds of fine-grained representations, region embedding and label distribution, respectively, from aspects of both feature and label. To optimize the region embedding, we propose a Fine-Grained Contrastive Learning (FGCL) strategy to simultaneously conduct class-level alignment and model the intra-class discrepancy. To optimize the label distribution, we propose a Fine-Grained Label Alignment (FGLA) strategy to align diverse fine-grained semantic information among samples, rather than merely label information. Finally, both region embedding and label distribution are utilized together to perform ZS-CMR at a finer granularity. Experimental results on three widely used datasets demonstrate that our method outperforms the state-of-the-art methods by a large margin. Detailed ablation studies have also been carried out, which provably affirm the advantage of each component we propose. Our code will be available at https://github.com/ShipingGe/FGAN .
Förderung
- National Natural Science Foundation of China
- Collaborative Innovation Center of Novel Software Technology and Industrialization, and Fundamental Research Funds for the Central Universities
Zitieren
APA
Ge, S., Jiang, Z., Yin, Y., Wang, C., Cheng, Z., & Gu, Q. (2025). Fine-Grained Alignment Network for Zero-Shot Cross-Modal Retrieval. ACM Transactions on Multimedia Computing, Communications, and Applications, 21(10), 1-24. https://doi.org/10.1145/3722223
Vancouver
Ge S, Jiang Z, Yin Y, Wang C, Cheng Z, Gu Q. Fine-Grained Alignment Network for Zero-Shot Cross-Modal Retrieval. ACM Transactions on Multimedia Computing, Communications, and Applications. 2025;21(10):1-24. doi:10.1145/3722223
BibTeX
@article{Ge2025,
author = {Ge, Shiping and Jiang, Zhiwei and Yin, Yafeng and Wang, Cong and Cheng, Zifeng and Gu, Qing},
title = {Fine-Grained Alignment Network for Zero-Shot Cross-Modal Retrieval},
journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
year = {2025},
volume = {21},
number = {10},
pages = {1--24},
doi = {10.1145/3722223}
}
EUVIMED