Improving Image-Text Matching With Bidirectional Consistency of Cross-Modal Alignment
IEEE Transactions on Circuits and Systems for Video Technology · 2024-07 · 34 (7) · 6590-6607
DOI: 10.1109/tcsvt.2024.3369656 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems for Video Technology, 2024-07. Verfasst von Zhe Li und 4 weiteren Autor:innen. Die Arbeit wurde bisher 44-mal zitiert (Crossref) und nennt 68 Literaturangaben. Gefördert von: National Outstanding Youth Science Fund Project of National Natural Science Foundation of China, Science Fund for Creative Research Groups.
Abstract (Originalsprache)
Image-text matching is a fundamental task in bridging the semantics between vision and language. The key challenge lies in establishing accurate alignment between two heterogeneous modalities. Existing cross-modal fine-grained matching methods normally include two alignment directions, “word to region” and “region to word”, and the overall image-text similarity is calculated from the alignments. However, the alignment of these two directions is typically independent, that is, the alignment of “word to region” and “region to word” is irrelevant, so the alignment consistency cannot be guaranteed in two directions, which inevitably introduces inconsistent alignments, leading to potential inaccurate image-text matching results. In this paper, we propose a novel Bidirectional cOnsistency netwOrks for cross-Modal alignment (BOOM), which achieves more accurate cross-modal semantic alignments by imposing explicit consistency constraints in both directions. Specifically, according to three aspects reflected by alignment consistency, i.e., significance, wholeness, and alignment orderliness, we design a novel systematic multi-granularity consistency constraints: point-wise consistency, which enforces consistency of the most significant single word item in bidirectional alignments; set-wise consistency, which maintains more comprehensive and accurate bidirectional entire alignment values consistent and order-wise consistency, which ensures order consistency of bidirectional alignment results. Bidirectional cross-modal alignment between words and regions is corrected from three different perspectives: maximum, distribution, and order. Extensive experiments on two benchmarks, i.e., Flickr30K and MS-COCO, demonstrate that our BOOM achieves state-of-the-art performance.
Förderung
- National Outstanding Youth Science Fund Project of National Natural Science Foundation of China
- Science Fund for Creative Research Groups
Zitieren
APA
Li, Z., Zhang, L., Zhang, K., Zhang, Y., & Mao, Z. (2024). Improving Image-Text Matching With Bidirectional Consistency of Cross-Modal Alignment. IEEE Transactions on Circuits and Systems for Video Technology, 34(7), 6590-6607. https://doi.org/10.1109/tcsvt.2024.3369656
Vancouver
Li Z, Zhang L, Zhang K, Zhang Y, Mao Z. Improving Image-Text Matching With Bidirectional Consistency of Cross-Modal Alignment. IEEE Transactions on Circuits and Systems for Video Technology. 2024;34(7):6590-6607. doi:10.1109/tcsvt.2024.3369656
BibTeX
@article{Li2024,
author = {Li, Zhe and Zhang, Lei and Zhang, Kun and Zhang, Yongdong and Mao, Zhendong},
title = {Improving Image-Text Matching With Bidirectional Consistency of Cross-Modal Alignment},
journal = {IEEE Transactions on Circuits and Systems for Video Technology},
year = {2024},
volume = {34},
number = {7},
pages = {6590--6607},
doi = {10.1109/tcsvt.2024.3369656}
}
EUVIMED