Action-Driven Semantic Representation and Aggregation for Video Captioning
IEEE Transactions on Circuits and Systems for Video Technology · 2025-04 · 35 (4) · 3383-3395
DOI: 10.1109/tcsvt.2024.3502736 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems for Video Technology, 2025-04. Verfasst von Tingting Han und 4 weiteren Autor:innen. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 51 Literaturangaben. Gefördert von: Zhejiang Province Natural Science Foundation, National Natural Science Foundation of China.
Abstract (Originalsprache)
Video captioning, a challenging task that entails generating natural language descriptions of visual content, often fails to effectively grasp the essence of action semantics. To harness the power of action detection to facilitate a deeper understanding of the video content, we propose an action-driven method, named Hierarchical Semantic Representation and Aggregation (HSRA) network. This method explicitly exploits action clues with a hierarchical semantic representation module, which models visual semantics in a three-level structure: “object-action-event”. By employing learnable action queries, our approach injects extensive action semantics into the model, thereby enabling more accurate and context-rich captions. To further enhance semantic alignment and understanding, we introduce a semantic aggregation composed of a semantic interaction module and a semantic refinement module. This component facilitates the alignment of semantics across different levels and emphasizes key information, ultimately leading to significant improvements in semantic consistency between the video and generated captions. We performed extensive evaluations on two well-established public datasets, MSVD and MSR-VTT, and the findings consistently demonstrate that our proposed HSRA network outperforms contemporary state-of-the-art methods.
Förderung
- Zhejiang Province Natural Science Foundation
- National Natural Science Foundation of China
Zitieren
APA
Han, T., Xu, Y., Yu, J., Yu, Z., & Zhao, S. (2025). Action-Driven Semantic Representation and Aggregation for Video Captioning. IEEE Transactions on Circuits and Systems for Video Technology, 35(4), 3383-3395. https://doi.org/10.1109/tcsvt.2024.3502736
Vancouver
Han T, Xu Y, Yu J, Yu Z, Zhao S. Action-Driven Semantic Representation and Aggregation for Video Captioning. IEEE Transactions on Circuits and Systems for Video Technology. 2025;35(4):3383-3395. doi:10.1109/tcsvt.2024.3502736
BibTeX
@article{Han2025,
author = {Han, Tingting and Xu, Yaochen and Yu, Jun and Yu, Zhou and Zhao, Sicheng},
title = {Action-Driven Semantic Representation and Aggregation for Video Captioning},
journal = {IEEE Transactions on Circuits and Systems for Video Technology},
year = {2025},
volume = {35},
number = {4},
pages = {3383--3395},
doi = {10.1109/tcsvt.2024.3502736}
}
EUVIMED