Learning Feature Semantic Matching for Spatio-Temporal Video Grounding
IEEE Transactions on Multimedia · 2024 · 26 · 9268-9279
DOI: 10.1109/tmm.2024.3387696 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Multimedia, 2024. Verfasst von Tong Zhang und 6 weiteren Autor:innen. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 60 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Major basic research project of Shandong Natural Science Foundation, Natural Science Foundation of Shandong Province.
Abstract (Originalsprache)
Spatio-temporal video grounding (STVG) aims to localize a spatio-temporal tube, including temporal boundaries and object bounding boxes, that semantically corresponds to a given language description in an untrimmed video. The existing one-stage solutions in this task face two significant challenges, namely, vision-text semantic misalignment and spatial mislocalization, which limit their performance in grounding. These two limitations are mainly caused by neglect of fine-grained alignment in cross-modality fusion and the reliance on a text-agnostic query in sequentially spatial localization. To address these issues, we propose an effective model with a newly designed Feature Semantic Matching (FSM) module based on a Transformer architecture to address the above issues. Our method introduces a cross-modal feature matching module to achieve multi-granularity alignment between video and text while preventing the weakening of important features during the feature fusion stage. Additionally, we design a query-modulated matching module to facilitate text-relevant tube construction by multiple query generation and tubulet sequence matching. To ensure the quality of tube construction, we employ a novel mismatching rectify contrastive loss to rectify the mismatching between the learnable query and the objects corresponding to the text descriptions by restricting the generated spatial query. Extensive experiments demonstrate that our method outperforms the state-of-the-art methods on two challenging STVG benchmarks.
Förderung
- National Natural Science Foundation of China
- Major basic research project of Shandong Natural Science Foundation
- Natural Science Foundation of Shandong Province
- Young Elite Scientists Sponsorship Program by CAST
- Shandong Provincial Natural Science Foundation for Distinguished Young Scholars
- Shandong Province Science and Technology Small and Medium-sized Enterprise Innovation Capacity Enhancement Project
- Shandong Province Higher Education Institutions Youth Entrepreneurship and Technology Support Program
- Taishan Scholar Project of Shandong Province
Zitieren
APA
Zhang, T., Fang, H., Zhang, H., Gao, J., Lu, X., Nie, X., & Yin, Y. (2024). Learning Feature Semantic Matching for Spatio-Temporal Video Grounding. IEEE Transactions on Multimedia, 26, 9268-9279. https://doi.org/10.1109/tmm.2024.3387696
Vancouver
Zhang T, Fang H, Zhang H, Gao J, Lu X, Nie X, et al. Learning Feature Semantic Matching for Spatio-Temporal Video Grounding. IEEE Transactions on Multimedia. 2024;26:9268-9279. doi:10.1109/tmm.2024.3387696
BibTeX
@article{Zhang2024,
author = {Zhang, Tong and Fang, Hao and Zhang, Hao and Gao, Jialin and Lu, Xiankai and Nie, Xiushan and Yin, Yilong},
title = {Learning Feature Semantic Matching for Spatio-Temporal Video Grounding},
journal = {IEEE Transactions on Multimedia},
year = {2024},
volume = {26},
pages = {9268--9279},
doi = {10.1109/tmm.2024.3387696}
}
EUVIMED