CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer
Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (6) · 5651-5659
DOI: 10.1609/aaai.v38i6.28376 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Yabing Wang und 3 weiteren Autor:innen. Die Arbeit wurde bisher 16-mal zitiert (Crossref).
Abstract (Originalsprache)
Cross-lingual cross-modal retrieval has garnered increasing attention recently, which aims to achieve the alignment between vision and target language (V-T) without using any annotated V-T data pairs. Current methods employ machine translation (MT) to construct pseudo-parallel data pairs, which are then used to learn a multi-lingual and multi-modal embedding space that aligns visual and target-language representations. However, the large heterogeneous gap between vision and text, along with the noise present in target language translations, poses significant challenges in effectively aligning their representations. To address these challenges, we propose a general framework, Cross-Lingual to Cross-Modal (CL2CM), which improves the alignment between vision and target language using cross-lingual transfer. This approach allows us to fully leverage the merits of multi-lingual pre-trained models (e.g., mBERT) and the benefits of the same modality structure, i.e., smaller gap, to provide reliable and comprehensive semantic correspondence (knowledge) for the cross-modal network. We evaluate our proposed approach on two multilingual image-text datasets, Multi30K and MSCOCO, and one video-text dataset, VATEX. The results clearly demonstrate the effectiveness of our proposed method and its high potential for large-scale retrieval.
Zitieren
APA
Wang, Y., Wang, F., Dong, J., & Luo, H. (2024). CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer. Proceedings of the AAAI Conference on Artificial Intelligence, 38(6), 5651-5659. https://doi.org/10.1609/aaai.v38i6.28376
Vancouver
Wang Y, Wang F, Dong J, Luo H. CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(6):5651-5659. doi:10.1609/aaai.v38i6.28376
BibTeX
@article{Wang2024,
author = {Wang, Yabing and Wang, Fan and Dong, Jianfeng and Luo, Hao},
title = {CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2024},
volume = {38},
number = {6},
pages = {5651--5659},
doi = {10.1609/aaai.v38i6.28376}
}
EUVIMED