EUVIMEDEUVIMED entdecken
Crossref

Improving Video Moment Retrieval by Auxiliary Moment-Query Pairs With Hyper-Interaction

Runhao Zeng, Yishen Zhuo, Jialiang Li, Yunjin Yang, Huisi Wu, Qi Chen, Xiping Hu, Victor C. M. Leung

IEEE Transactions on Circuits and Systems for Video Technology · 2025-05 · 35 (5) · 3940-3954

DOI: 10.1109/tcsvt.2024.3513633 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems for Video Technology, 2025-05. Verfasst von Runhao Zeng und 7 weiteren Autor:innen. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 72 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Shenzhen Natural Science Foundation, Excellent Science and Technology Creative Talent Training Program of Shenzhen Municipality.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Most existing video moment retrieval (VMR) benchmark datasets face a common issue of sparse annotations-only a few moments being annotated. We argue that videos contain a broader range of meaningful moments that, if leveraged, could significantly enhance performance. Existing methods typically follow a generate-then-select paradigm, focusing primarily on generating moment-query pairs while neglecting the crucial aspect of selection. In this paper, we propose a new method, HyperAux, to yield auxiliary moment-query pairs by modeling the multi-modal hyper-interaction between video and language. Specifically, given a set of candidate moment-query pairs from a video, we construct a hypergraph with multiple hyperedges, each corresponding to a moment-query pair. Unlike traditional graphs where each edge connects only two nodes (frames or queries), each hyperedge connects multiple nodes, including all frames within a moment, semantically related frames outside the moment, and an input query. This design allows us to consider the frames within a moment as a whole, rather than modeling individual frame-query relationships separately. More importantly, constructing the relationships among all moment-query pairs within a video into a large hypergraph facilitates selecting higher-quality data from such pairs. On this hypergraph, we employ a hypergraph neural network to aggregate node information, update the hyperedge, and propagate video-language hyper-interactions to each connected node, resulting in context-aware node representations. This enables us to use node relevance to select high-quality moment-query pairs and refine the moments’ boundaries. We also exploit the discrepancy in semantic matching within and outside moments to construct a loss function for training the HGNN without human annotations. Our auxiliary data enhances the performance of twelve VMR models under fully-supervised, weakly-supervised, and zero-shot settings across three widely used VMR datasets: ActivityNet Captions, Charades-STA, and QVHighlights. We will release the source code and models publicly.

Förderung

  • National Natural Science Foundation of China
  • Shenzhen Natural Science Foundation
  • Excellent Science and Technology Creative Talent Training Program of Shenzhen Municipality
  • Basic and Applied Basic Research Foundation of Guangdong Province
  • Key Scientific Research Project of the Department of Education of Guangdong Province
  • Guangdong-Hong Kong-Macao Joint Laboratory for Emotional Intelligence and Pervasive Computing

Zitieren

APA

Zeng, R., Zhuo, Y., Li, J., Yang, Y., Wu, H., Chen, Q., Hu, X., & Leung, V. C. M. (2025). Improving Video Moment Retrieval by Auxiliary Moment-Query Pairs With Hyper-Interaction. IEEE Transactions on Circuits and Systems for Video Technology, 35(5), 3940-3954. https://doi.org/10.1109/tcsvt.2024.3513633

Vancouver

Zeng R, Zhuo Y, Li J, Yang Y, Wu H, Chen Q, et al. Improving Video Moment Retrieval by Auxiliary Moment-Query Pairs With Hyper-Interaction. IEEE Transactions on Circuits and Systems for Video Technology. 2025;35(5):3940-3954. doi:10.1109/tcsvt.2024.3513633

BibTeX
@article{Zeng2025,
  author = {Zeng, Runhao and Zhuo, Yishen and Li, Jialiang and Yang, Yunjin and Wu, Huisi and Chen, Qi and Hu, Xiping and Leung, Victor C. M.},
  title = {Improving Video Moment Retrieval by Auxiliary Moment-Query Pairs With Hyper-Interaction},
  journal = {IEEE Transactions on Circuits and Systems for Video Technology},
  year = {2025},
  volume = {35},
  number = {5},
  pages = {3940--3954},
  doi = {10.1109/tcsvt.2024.3513633}
}
Kostenlos starten →