SEMScene: Semantic-Consistency Enhanced Multi-Level Scene Graph Matching for Image-Text Retrieval
ACM Transactions on Multimedia Computing, Communications, and Applications · 31.08.2024 · 20 (8) · 1-28
DOI: 10.1145/3664816 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 16.08.2024. Verfasst von Yuankun Liu und 7 weiteren Autor:innen. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 50 Literaturangaben. Gefördert von: National Key R&D Program of China.
Abstract (Originalsprache)
Image-text retrieval, a fundamental cross-modal task, performs similarity reasoning for images and texts. The primary challenge for image-text retrieval is cross-modal semantic heterogeneity, where the semantic features of visual and textual modalities are rich but distinct. Scene graph is an effective representation for images and texts as it explicitly models objects and their relations. Existing scene graph based methods have not fully taken the features regarding various granularities implicit in scene graph into consideration (e.g., triplets), the inadequate feature matching incurs the absence of non-trivial semantic information (e.g., inner relations among triplets). Therefore, we propose a S emantic-Consistency E nhanced M ulti-Level Scene Graph Matching (SEMScene) network, which exploits the semantic relevance between visual and textual scene graphs from fine-grained to coarse-grained. Firstly, under the scene graph representation, we perform feature matching including low-level node matching, mid-level semantic triplet matching, and high-level holistic scene graph matching. Secondly, to enhance the semantic-consistency for object-fused triplets carrying key correlation information, we propose a dual-step constraint mechanism in mid-level matching. Thirdly, to guide the model to learn the semantic-consistency of matched image-text pairs, we devise effective loss functions for each stage of the dual-step constraint. Comprehensive experiments on Flickr30K and MS-COCO datasets demonstrate that SEMScene achieves state-of-the-art performances with significant improvements.
Förderung
- National Key R&D Program of China
Zitieren
APA
Liu, Y., Yuan, X., Li, H., Tan, Z., Huang, J., Xiao, J., Li, W., & Mo, T. (2024). SEMScene: Semantic-Consistency Enhanced Multi-Level Scene Graph Matching for Image-Text Retrieval. ACM Transactions on Multimedia Computing, Communications, and Applications, 20(8), 1-28. https://doi.org/10.1145/3664816
Vancouver
Liu Y, Yuan X, Li H, Tan Z, Huang J, Xiao J, et al. SEMScene: Semantic-Consistency Enhanced Multi-Level Scene Graph Matching for Image-Text Retrieval. ACM Transactions on Multimedia Computing, Communications, and Applications. 2024;20(8):1-28. doi:10.1145/3664816
BibTeX
@article{Liu2024,
author = {Liu, Yuankun and Yuan, Xiang and Li, Haochen and Tan, Zhijie and Huang, Jinsong and Xiao, Jingjie and Li, Weiping and Mo, Tong},
title = {SEMScene: Semantic-Consistency Enhanced Multi-Level Scene Graph Matching for Image-Text Retrieval},
journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
year = {2024},
volume = {20},
number = {8},
pages = {1--28},
doi = {10.1145/3664816}
}
EUVIMED