Refined Semantic Enhancement towards Frequency Diffusion for Video Captioning
Proceedings of the AAAI Conference on Artificial Intelligence · 26.06.2023 · 37 (3) · 3724-3732
DOI: 10.1609/aaai.v37i3.25484 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 26.06.2023. Verfasst von Xian Zhong und 5 weiteren Autor:innen. Die Arbeit wurde bisher 54-mal zitiert (Crossref).
Abstract (Originalsprache)
Video captioning aims to generate natural language sentences that describe the given video accurately. Existing methods obtain favorable generation by exploring richer visual representations in encode phase or improving the decoding ability. However, the long-tailed problem hinders these attempts at low-frequency tokens, which rarely occur but carry critical semantics, playing a vital role in the detailed generation. In this paper, we introduce a novel Refined Semantic enhancement method towards Frequency Diffusion (RSFD), a captioning model that constantly perceives the linguistic representation of the infrequent tokens. Concretely, a Frequency-Aware Diffusion (FAD) module is proposed to comprehend the semantics of low-frequency tokens to break through generation limitations. In this way, the caption is refined by promoting the absorption of tokens with insufficient occurrence. Based on FAD, we design a Divergent Semantic Supervisor (DSS) module to compensate for the information loss of high-frequency tokens brought by the diffusion process, where the semantics of low-frequency tokens is further emphasized to alleviate the long-tailed problem. Extensive experiments indicate that RSFD outperforms the state-of-the-art methods on two benchmark datasets, i.e., MSR-VTT and MSVD, demonstrate that the enhancement of low-frequency tokens semantics can obtain a competitive generation effect. Code is available at https://github.com/lzp870/RSFD.
Zitieren
APA
Zhong, X., Li, Z., Chen, S., Jiang, K., Chen, C., & Ye, M. (2023). Refined Semantic Enhancement towards Frequency Diffusion for Video Captioning. Proceedings of the AAAI Conference on Artificial Intelligence, 37(3), 3724-3732. https://doi.org/10.1609/aaai.v37i3.25484
Vancouver
Zhong X, Li Z, Chen S, Jiang K, Chen C, Ye M. Refined Semantic Enhancement towards Frequency Diffusion for Video Captioning. Proceedings of the AAAI Conference on Artificial Intelligence. 2023;37(3):3724-3732. doi:10.1609/aaai.v37i3.25484
BibTeX
@article{Zhong2023,
author = {Zhong, Xian and Li, Zipeng and Chen, Shuqin and Jiang, Kui and Chen, Chen and Ye, Mang},
title = {Refined Semantic Enhancement towards Frequency Diffusion for Video Captioning},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2023},
volume = {37},
number = {3},
pages = {3724--3732},
doi = {10.1609/aaai.v37i3.25484}
}
EUVIMED