EUVIMEDEUVIMED entdecken
Crossref

Unpaired Image-Text Matching via Multimodal Aligned Conceptual Knowledge

Yan Huang, Yuming Wang, Yunan Zeng, Junshi Huang, Zhenhua Chai, Liang Wang

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-07 · 47 (7) · 5160-5176

DOI: 10.1109/tpami.2024.3432552 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-07. Verfasst von Yan Huang und 5 weiteren Autor:innen. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 69 Literaturangaben. Gefördert von: National Key R&D Program of China, National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recently, the accuracy of image-text matching has been greatly improved by multimodal pretrained models, all of which use millions or billions of paired images and texts for supervised model learning. Different from them, human brains can well match images with texts using their stored multimodal knowledge. Inspired by that, this paper studies a new scenario as unpaired image-text matching, in which paired images and texts are assumed to be unavailable during model learning. To deal with it, we accordingly propose a simple yet effective method namely Multimodal Aligned Conceptual Knowledge (MACK). First, we collect a set of words and their related image regions from publicly available datasets, and compute prototypical region representations to obtain pretrained general knowledge. To make the obtained knowledge better suit for certain datasets, we refine it using unpaired images and texts in a self-supervised learning manner to obtain fine-tuned domain knowledge. Then, to match given images with texts based on the knowledge, we represent parsed words in the texts by prototypical region representations, and compute region-word similarity scores. At last, the scores are aggregated based on bidirectional similarity pooling into an image-text similarity score, which can be directly used for unpaired image-text matching. The proposed MACK is complementary with existing models, which can be easily extended as a re-ranking method to substantially improve their performance of zero-shot and cross-dataset image-text matching.

Förderung

  • National Key R&D Program of China
  • National Natural Science Foundation of China

Zitieren

APA

Huang, Y., Wang, Y., Zeng, Y., Huang, J., Chai, Z., & Wang, L. (2025). Unpaired Image-Text Matching via Multimodal Aligned Conceptual Knowledge. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(7), 5160-5176. https://doi.org/10.1109/tpami.2024.3432552

Vancouver

Huang Y, Wang Y, Zeng Y, Huang J, Chai Z, Wang L. Unpaired Image-Text Matching via Multimodal Aligned Conceptual Knowledge. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(7):5160-5176. doi:10.1109/tpami.2024.3432552

BibTeX
@article{Huang2025,
  author = {Huang, Yan and Wang, Yuming and Zeng, Yunan and Huang, Junshi and Chai, Zhenhua and Wang, Liang},
  title = {Unpaired Image-Text Matching via Multimodal Aligned Conceptual Knowledge},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2025},
  volume = {47},
  number = {7},
  pages = {5160--5176},
  doi = {10.1109/tpami.2024.3432552}
}
Kostenlos starten →