EUVIMEDEUVIMED entdecken
Crossref

Enhancing Video-Language Representations With Structural Spatio-Temporal Alignment

Hao Fei, Shengqiong Wu, Meishan Zhang, Min Zhang, Tat-Seng Chua, Shuicheng Yan

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-12 · 46 (12) · 7701-7719

DOI: 10.1109/tpami.2024.3393452 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-12. Verfasst von Hao Fei und 5 weiteren Autor:innen. Die Arbeit wurde bisher 50-mal zitiert (Crossref) und nennt 81 Literaturangaben. Gefördert von: National Natural Science Foundation of China, CCF-Baidu Open Fund, NExT Research Center.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

While pre-training large-scale video-language models (VLMs) has shown remarkable potential for various downstream video-language tasks, existing VLMs can still suffer from certain commonly seen limitations, e.g., coarse-grained cross-modal aligning, under-modeling of temporal dynamics, detached video-language view. In this work, we target enhancing VLMs with a fine-grained structural spatio-temporal alignment learning method (namely Finsta). First of all, we represent the input texts and videos with fine-grained scene graph (SG) structures, both of which are further unified into a holistic SG (HSG) for bridging two modalities. Then, an SG-based framework is built, where the textual SG (TSG) is encoded with a graph Transformer, while the video dynamic SG (DSG) and the HSG are modeled with a novel recurrent graph Transformer for spatial and temporal feature propagation. A spatial-temporal Gaussian differential graph Transformer is further devised to strengthen the sense of the changes in objects across spatial and temporal dimensions. Next, based on the fine-grained structural features of TSG and DSG, we perform object-centered spatial alignment and predicate-centered temporal alignment respectively, enhancing the video-language grounding in both the spatiality and temporality. We design our method as a plug&play system, which can be integrated into existing well-trained VLMs for further representation augmentation, without training from scratch or relying on SG annotations in downstream applications. On 6 representative VL modeling tasks over 12 datasets in both standard and long-form video scenarios, Finsta consistently improves the existing 13 strong-performing VLMs persistently, and refreshes the current state-of-the-art end task performance significantly in both the fine-tuning and zero-shot settings.

Förderung

  • National Natural Science Foundation of China
  • CCF-Baidu Open Fund
  • NExT Research Center
  • Skywork AI, Singapore

Zitieren

APA

Fei, H., Wu, S., Zhang, M., Zhang, M., Chua, T. S., & Yan, S. (2024). Enhancing Video-Language Representations With Structural Spatio-Temporal Alignment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12), 7701-7719. https://doi.org/10.1109/tpami.2024.3393452

Vancouver

Fei H, Wu S, Zhang M, Zhang M, Chua TS, Yan S. Enhancing Video-Language Representations With Structural Spatio-Temporal Alignment. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(12):7701-7719. doi:10.1109/tpami.2024.3393452

BibTeX
@article{Fei2024,
  author = {Fei, Hao and Wu, Shengqiong and Zhang, Meishan and Zhang, Min and Chua, Tat-Seng and Yan, Shuicheng},
  title = {Enhancing Video-Language Representations With Structural Spatio-Temporal Alignment},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2024},
  volume = {46},
  number = {12},
  pages = {7701--7719},
  doi = {10.1109/tpami.2024.3393452}
}
Kostenlos starten →