Learning Commonsense-aware Moment-Text Alignment for Fast Video Temporal Grounding
ACM Transactions on Multimedia Computing, Communications, and Applications · 31.10.2024 · 20 (10) · 1-22
DOI: 10.1145/3663368 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 12.09.2024. Verfasst von Ziyue Wu und 3 weiteren Autor:innen. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 89 Literaturangaben. Gefördert von: National Key Research and Development Plan of China, National Natural Science Foundation of China.
Abstract (Originalsprache)
Grounding temporal video segments described in natural language queries effectively and efficiently is a crucial capability needed in vision-and-language fields. In this article, we deal with the fast video temporal grounding (FVTG) task, aiming at localizing the target segment with high speed and favorable accuracy. Most existing approaches adopt elaborately designed cross-modal interaction modules to improve the grounding performance, which suffer from the test-time bottleneck. Although several common space-based methods enjoy the high-speed merit during inference, they can hardly capture the comprehensive and explicit relations between visual and textual modalities. In this article, to tackle the dilemma of the speed–accuracy tradeoff, we propose a commonsense-aware cross-modal alignment network (C 2 AN) that incorporates commonsense-guided visual and text representations into a complementary common space for fast video temporal grounding. Specifically, the commonsense concepts are explored and exploited by extracting the structural semantic information from a language corpus. Then, a commonsense-aware interaction module is designed to obtain bridged visual and text features by utilizing the learned commonsense concepts. Finally, to maintain the original semantic information of textual queries, a cross-modal complementary common space is optimized to obtain matching scores for performing FVTG. Extensive results on two challenging benchmarks show that our C 2 AN method performs favorably against states of the art while running at high speed. Our code is available at https://github.com/ZiyueWu59/CCA
Förderung
- National Key Research and Development Plan of China
- National Natural Science Foundation of China
Zitieren
APA
Wu, Z., Gao, J., Huang, S., & Xu, C. (2024). Learning Commonsense-aware Moment-Text Alignment for Fast Video Temporal Grounding. ACM Transactions on Multimedia Computing, Communications, and Applications, 20(10), 1-22. https://doi.org/10.1145/3663368
Vancouver
Wu Z, Gao J, Huang S, Xu C. Learning Commonsense-aware Moment-Text Alignment for Fast Video Temporal Grounding. ACM Transactions on Multimedia Computing, Communications, and Applications. 2024;20(10):1-22. doi:10.1145/3663368
BibTeX
@article{Wu2024,
author = {Wu, Ziyue and Gao, Junyu and Huang, Shucheng and Xu, Changsheng},
title = {Learning Commonsense-aware Moment-Text Alignment for Fast Video Temporal Grounding},
journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
year = {2024},
volume = {20},
number = {10},
pages = {1--22},
doi = {10.1145/3663368}
}
EUVIMED