A Coding Framework and Benchmark Towards Low-Bitrate Video Understanding
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-08 · 46 (8) · 5852-5872
DOI: 10.1109/tpami.2024.3367879 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-08. Verfasst von Yuan Tian und 5 weiteren Autor:innen. Die Arbeit wurde bisher 45-mal zitiert (Crossref) und nennt 137 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Fundamental Research Funds for the Central Universities, National Key R&D Program of China.
Abstract (Originalsprache)
Video compression is indispensable to most video analysis systems. Despite saving the transportation bandwidth, it also deteriorates downstream video understanding tasks, especially at low-bitrate settings. To systematically investigate this problem, we first thoroughly review the previous methods, revealing that three principles, i.e., task-decoupled, label-free, and data-emerged semantic prior, are critical to a machine-friendly coding framework but are not fully satisfied so far. In this paper, we propose a traditional-neural mixed coding framework that simultaneously fulfills all these principles, by taking advantage of both traditional codecs and neural networks (NNs). On one hand, the traditional codecs can efficiently encode the pixel signal of videos but may distort the semantic information. On the other hand, highly non-linear NNs are proficient in condensing video semantics into a compact representation. The framework is optimized by ensuring that a transportation-efficient semantic representation of the video is preserved w.r.t. the coding procedure, which is spontaneously learned from unlabeled data in a self-supervised manner. The videos collaboratively decoded from two streams (codec and NN) are of rich semantics, as well as visually photo-realistic, empirically boosting several mainstream downstream video analysis task performances without any post-adaptation procedure. Furthermore, by introducing the attention mechanism and adaptive modeling scheme, the video semantic modeling ability of our approach is further enhanced. Fianlly, we build a low-bitrate video understanding benchmark with three downstream tasks on eight datasets, demonstrating the notable superiority of our approach. All codes, data, and models will be open-sourced for facilitating future research.
Förderung
- National Natural Science Foundation of China
- Fundamental Research Funds for the Central Universities
- National Key R&D Program of China
- Shanghai Municipal Science and Technology
- Science and Technology Commission of Shanghai Municipality
Zitieren
APA
Tian, Y., Lu, G., Yan, Y., Zhai, G., Chen, L., & Gao, Z. (2024). A Coding Framework and Benchmark Towards Low-Bitrate Video Understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(8), 5852-5872. https://doi.org/10.1109/tpami.2024.3367879
Vancouver
Tian Y, Lu G, Yan Y, Zhai G, Chen L, Gao Z. A Coding Framework and Benchmark Towards Low-Bitrate Video Understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(8):5852-5872. doi:10.1109/tpami.2024.3367879
BibTeX
@article{Tian2024,
author = {Tian, Yuan and Lu, Guo and Yan, Yichao and Zhai, Guangtao and Chen, Li and Gao, Zhiyong},
title = {A Coding Framework and Benchmark Towards Low-Bitrate Video Understanding},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2024},
volume = {46},
number = {8},
pages = {5852--5872},
doi = {10.1109/tpami.2024.3367879}
}
EUVIMED