CoSTA: End-to-End Comprehensive Space-Time Entanglement for Spatio-Temporal Video Grounding
Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (4) · 3324-3332
DOI: 10.1609/aaai.v38i4.28118 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Yaoyuan Liang und 7 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref).
Abstract (Originalsprache)
This paper studies the spatio-temporal video grounding task, which aims to localize a spatio-temporal tube in an untrimmed video based on the given text description of an event. Existing one-stage approaches suffer from insufficient space-time interaction in two aspects: i) less precise prediction of event temporal boundaries, and ii) inconsistency in object prediction for the same event across adjacent frames. To address these issues, we propose a framework of Comprehensive Space-Time entAnglement (CoSTA) to densely entangle space-time multi-modal features for spatio-temporal localization. Specifically, we propose a space-time collaborative encoder to extract comprehensive video features and leverage Transformer to perform spatio-temporal multi-modal understanding. Our entangled decoder couples temporal boundary prediction and spatial localization via an entangled query, boasting an enhanced ability to capture object-event relationships. We conduct extensive experiments on the challenging benchmarks of HC-STVG and VidSTG, where CoSTA outperforms existing state-of-the-art methods, demonstrating its effectiveness for this task.
Zitieren
APA
Liang, Y., Liang, X., Tang, Y., Yang, Z., Li, Z., Wang, J., Ding, W., & Huang, S. (2024). CoSTA: End-to-End Comprehensive Space-Time Entanglement for Spatio-Temporal Video Grounding. Proceedings of the AAAI Conference on Artificial Intelligence, 38(4), 3324-3332. https://doi.org/10.1609/aaai.v38i4.28118
Vancouver
Liang Y, Liang X, Tang Y, Yang Z, Li Z, Wang J, et al. CoSTA: End-to-End Comprehensive Space-Time Entanglement for Spatio-Temporal Video Grounding. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(4):3324-3332. doi:10.1609/aaai.v38i4.28118
BibTeX
@article{Liang2024,
author = {Liang, Yaoyuan and Liang, Xiao and Tang, Yansong and Yang, Zhao and Li, Ziran and Wang, Jingang and Ding, Wenbo and Huang, Shao-Lun},
title = {CoSTA: End-to-End Comprehensive Space-Time Entanglement for Spatio-Temporal Video Grounding},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2024},
volume = {38},
number = {4},
pages = {3324--3332},
doi = {10.1609/aaai.v38i4.28118}
}
EUVIMED