MGSGA: Multi-grained and Semantic-Guided Alignment for Text-Video Retrieval
Neural Processing Letters · 17.02.2024 · 56 (2)
DOI: 10.1007/s11063-024-11468-5 ↗Kurz gefasst
Fachartikel in der Zeitschrift Neural Processing Letters, online erschienen am 17.02.2024. Verfasst von Xiaoyu Wu und 2 weiteren Autor:innen. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 54 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Key Research and Development Program of China, National Natural Science Foundation of China, Construction Project of “Beijing top notch”Discipline.
Abstract (Originalsprache)
In the text-video retrieval task, the objective is to calculate the similarity between a text and a video, and rank the relevant candidates higher. Most existing methods only consider the text-video semantic alignment in the global view. But using mean-pooling to obtain global semantics and simply aligning text and video in the global view may lead to semantic bias. In addition, some methods utilize offline object detectors or sentence parsers to obtain entity-level information in text and video and achieve local alignment. However, inaccurate detection introduces possible errors and such approaches prevent models from being trained end-to-end for retrieval. To overcome these limitations, we propose multi-grained and semantic-guided alignment for text-video retrieval in this paper, which can achieve fine-grained alignment based on video frames and text words, local alignment based on semantic centers, and global alignment. Specially, we explore summary semantics of text and video to guide the local alignment based on semantic centers for we believe that the importance of each semantic center is determined by summary semantics. We evaluate our approach on four benchmark datasets of MSRVTT, MSVD, ActivityNet Captions, and DiDeMo, achieving better performance than most existing methods.
Förderung
- National Key Research and Development Program of China
- National Natural Science Foundation of China
- Construction Project of “Beijing top notch”Discipline
Zitieren
APA
Wu, X., Qian, J., & Yang, L. (2024). MGSGA: Multi-grained and Semantic-Guided Alignment for Text-Video Retrieval. Neural Processing Letters, 56(2). https://doi.org/10.1007/s11063-024-11468-5
Vancouver
Wu X, Qian J, Yang L. MGSGA: Multi-grained and Semantic-Guided Alignment for Text-Video Retrieval. Neural Processing Letters. 2024;56(2). doi:10.1007/s11063-024-11468-5
BibTeX
@article{Wu2024,
author = {Wu, Xiaoyu and Qian, Jiayao and Yang, Lulu},
title = {MGSGA: Multi-grained and Semantic-Guided Alignment for Text-Video Retrieval},
journal = {Neural Processing Letters},
year = {2024},
volume = {56},
number = {2},
doi = {10.1007/s11063-024-11468-5}
}
EUVIMED