EUVIMEDEUVIMED entdecken
Crossref

Learning Hierarchical Modular Networks for Video Captioning

Guorong Li, Hanhua Ye, Yuankai Qi, Shuhui Wang, Laiyun Qing, Qingming Huang, Ming-Hsuan Yang

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-02 · 46 (2) · 1049-1064

DOI: 10.1109/tpami.2023.3327677 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-02. Verfasst von Guorong Li und 6 weiteren Autor:innen. Die Arbeit wurde bisher 40-mal zitiert (Crossref) und nennt 87 Literaturangaben. Gefördert von: National Key R&D Program of China, National Natural Science Foundation of China, Chinese Academy of Sciences.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Video captioning aims to generate natural language descriptions for a given video clip. Existing methods mainly focus on end-to-end representation learning via word-by-word comparison between predicted captions and ground-truth texts. Although significant progress has been made, such supervised approaches neglect semantic alignment between visual and linguistic entities, which may negatively affect the generated captions. In this work, we propose a hierarchical modular network to bridge video representations and linguistic semantics at four granularities before generating captions: entity, verb, predicate, and sentence. Each level is implemented by one module to embed corresponding semantics into video representations. Additionally, we present a reinforcement learning module based on the scene graph of captions to better measure sentence similarity. Extensive experimental results show that the proposed method performs favorably against the state-of-the-art models on three widely-used benchmark datasets, including microsoft research video description corpus (MSVD), MSR-video to text (MSR-VTT), and video-and-TEXt (VATEX).

Förderung

  • National Key R&D Program of China
  • National Natural Science Foundation of China
  • Chinese Academy of Sciences
  • Fundamental Research Funds for Central Universities

Zitieren

APA

Li, G., Ye, H., Qi, Y., Wang, S., Qing, L., Huang, Q., & Yang, M. H. (2024). Learning Hierarchical Modular Networks for Video Captioning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(2), 1049-1064. https://doi.org/10.1109/tpami.2023.3327677

Vancouver

Li G, Ye H, Qi Y, Wang S, Qing L, Huang Q, et al. Learning Hierarchical Modular Networks for Video Captioning. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(2):1049-1064. doi:10.1109/tpami.2023.3327677

BibTeX
@article{Li2024,
  author = {Li, Guorong and Ye, Hanhua and Qi, Yuankai and Wang, Shuhui and Qing, Laiyun and Huang, Qingming and Yang, Ming-Hsuan},
  title = {Learning Hierarchical Modular Networks for Video Captioning},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2024},
  volume = {46},
  number = {2},
  pages = {1049--1064},
  doi = {10.1109/tpami.2023.3327677}
}
Kostenlos starten →