Breaking Through the Noisy Correspondence: A Robust Model for Image-Text Matching
ACM Transactions on Information Systems · 30.11.2024 · 42 (6) · 1-26
DOI: 10.1145/3662732 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Information Systems, online erschienen am 19.08.2024. Verfasst von Haitao Shi und 5 weiteren Autor:innen. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 61 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Shandong Provincial Natural Science Foundation, Science and Technology Innovation Program for Distinguished Young Scholars of Shandong Province Higher Education Institutions.
Abstract (Originalsprache)
Unleashing the power of image-text matching in real-world applications is hampered by noisy correspondence. Manually curating high-quality datasets is expensive and time-consuming, and datasets generated using diffusion models are not adequately well-aligned. The most promising way is to collect image-text pairs from the Internet, but it will inevitably introduce noisy correspondence. To reduce the negative impact of noisy correspondence, we propose a novel model that first transforms the noisy correspondence filtering problem into a similarity distribution modeling problem by exploiting the powerful capabilities of pre-trained models. Specifically, we use the Gaussian Mixture model to model the similarity obtained by CLIP as clean distribution and noisy distribution, to filter out most of the noisy correspondence in the dataset. Afterward, we used relatively clean data to fine-tune the model. To further reduce the negative impact of unfiltered noisy correspondence, i.e., a minimal part where two distributions intersect during the fine-tuning process, we propose a distribution-sensitive dynamic margin ranking loss, further increasing the distance between the two distributions. Through continuous iteration, the noisy correspondence gradually decreases and the model performance gradually improves. Our extensive experiments demonstrate the effectiveness and robustness of our model even under high noise rates.
Förderung
- National Natural Science Foundation of China
- Shandong Provincial Natural Science Foundation
- Science and Technology Innovation Program for Distinguished Young Scholars of Shandong Province Higher Education Institutions
Zitieren
APA
Shi, H., Liu, M., Mu, X., Song, X., Hu, Y., & Nie, L. (2024). Breaking Through the Noisy Correspondence: A Robust Model for Image-Text Matching. ACM Transactions on Information Systems, 42(6), 1-26. https://doi.org/10.1145/3662732
Vancouver
Shi H, Liu M, Mu X, Song X, Hu Y, Nie L. Breaking Through the Noisy Correspondence: A Robust Model for Image-Text Matching. ACM Transactions on Information Systems. 2024;42(6):1-26. doi:10.1145/3662732
BibTeX
@article{Shi2024,
author = {Shi, Haitao and Liu, Meng and Mu, Xiaoxuan and Song, Xuemeng and Hu, Yupeng and Nie, Liqiang},
title = {Breaking Through the Noisy Correspondence: A Robust Model for Image-Text Matching},
journal = {ACM Transactions on Information Systems},
year = {2024},
volume = {42},
number = {6},
pages = {1--26},
doi = {10.1145/3662732}
}
EUVIMED