Learning Hierarchical Modular Networks for Video Captioning
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-02 · 46 (2) · 1049-1064
DOI: 10.1109/tpami.2023.3327677 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-02. Verfasst von Guorong Li und 6 weiteren Autor:innen. Die Arbeit wurde bisher 40-mal zitiert (Crossref) und nennt 87 Literaturangaben. Gefördert von: National Key R&D Program of China, National Natural Science Foundation of China, Chinese Academy of Sciences.
Abstract (Originalsprache)
Video captioning aims to generate natural language descriptions for a given video clip. Existing methods mainly focus on end-to-end representation learning via word-by-word comparison between predicted captions and ground-truth texts. Although significant progress has been made, such supervised approaches neglect semantic alignment between visual and linguistic entities, which may negatively affect the generated captions. In this work, we propose a hierarchical modular network to bridge video representations and linguistic semantics at four granularities before generating captions: entity, verb, predicate, and sentence. Each level is implemented by one module to embed corresponding semantics into video representations. Additionally, we present a reinforcement learning module based on the scene graph of captions to better measure sentence similarity. Extensive experimental results show that the proposed method performs favorably against the state-of-the-art models on three widely-used benchmark datasets, including microsoft research video description corpus (MSVD), MSR-video to text (MSR-VTT), and video-and-TEXt (VATEX).
Förderung
- National Key R&D Program of China
- National Natural Science Foundation of China
- Chinese Academy of Sciences
- Fundamental Research Funds for Central Universities
Zitieren
APA
Li, G., Ye, H., Qi, Y., Wang, S., Qing, L., Huang, Q., & Yang, M. H. (2024). Learning Hierarchical Modular Networks for Video Captioning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(2), 1049-1064. https://doi.org/10.1109/tpami.2023.3327677
Vancouver
Li G, Ye H, Qi Y, Wang S, Qing L, Huang Q, et al. Learning Hierarchical Modular Networks for Video Captioning. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(2):1049-1064. doi:10.1109/tpami.2023.3327677
BibTeX
@article{Li2024,
author = {Li, Guorong and Ye, Hanhua and Qi, Yuankai and Wang, Shuhui and Qing, Laiyun and Huang, Qingming and Yang, Ming-Hsuan},
title = {Learning Hierarchical Modular Networks for Video Captioning},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2024},
volume = {46},
number = {2},
pages = {1049--1064},
doi = {10.1109/tpami.2023.3327677}
}
EUVIMED