EUVIMEDEUVIMED entdecken
Crossref

CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer

Yabing Wang, Fan Wang, Jianfeng Dong, Hao Luo

Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (6) · 5651-5659

DOI: 10.1609/aaai.v38i6.28376 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Yabing Wang und 3 weiteren Autor:innen. Die Arbeit wurde bisher 16-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Cross-lingual cross-modal retrieval has garnered increasing attention recently, which aims to achieve the alignment between vision and target language (V-T) without using any annotated V-T data pairs. Current methods employ machine translation (MT) to construct pseudo-parallel data pairs, which are then used to learn a multi-lingual and multi-modal embedding space that aligns visual and target-language representations. However, the large heterogeneous gap between vision and text, along with the noise present in target language translations, poses significant challenges in effectively aligning their representations. To address these challenges, we propose a general framework, Cross-Lingual to Cross-Modal (CL2CM), which improves the alignment between vision and target language using cross-lingual transfer. This approach allows us to fully leverage the merits of multi-lingual pre-trained models (e.g., mBERT) and the benefits of the same modality structure, i.e., smaller gap, to provide reliable and comprehensive semantic correspondence (knowledge) for the cross-modal network. We evaluate our proposed approach on two multilingual image-text datasets, Multi30K and MSCOCO, and one video-text dataset, VATEX. The results clearly demonstrate the effectiveness of our proposed method and its high potential for large-scale retrieval.

Zitieren

APA

Wang, Y., Wang, F., Dong, J., & Luo, H. (2024). CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer. Proceedings of the AAAI Conference on Artificial Intelligence, 38(6), 5651-5659. https://doi.org/10.1609/aaai.v38i6.28376

Vancouver

Wang Y, Wang F, Dong J, Luo H. CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(6):5651-5659. doi:10.1609/aaai.v38i6.28376

BibTeX
@article{Wang2024,
  author = {Wang, Yabing and Wang, Fan and Dong, Jianfeng and Luo, Hao},
  title = {CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2024},
  volume = {38},
  number = {6},
  pages = {5651--5659},
  doi = {10.1609/aaai.v38i6.28376}
}
Kostenlos starten →