Ultra-Low Bitrate Face Video Compression Based on Conversions From 3D Keypoints to 2D Motion Map
IEEE Transactions on Image Processing · 2024 · 33 · 6850-6864
DOI: 10.1109/tip.2024.3518100 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Image Processing, 2024. Verfasst von Zhao Wang und 5 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 77 Literaturangaben. Gefördert von: “Leading Goose” Research and Development Program of Zhejiang Province, National Natural Science Foundation of China, Fundamental Research Funds for the Central Universities.
Abstract (Originalsprache)
How to compress face video is a crucial problem for a series of online applications, such as video chat/conference, live broadcasting and remote education. Compared to other natural videos, these face-centric videos owning abundant structural information can be compactly represented and high-quality reconstructed via deep generative models, such that the promising compression performance can be achieved. However, the existing generative face video compression schemes are faced with the inconsistency between the 3D facial motion in the physical world and the face content evolution in the 2D view. To solve this drawback, we propose a 3D-Keypoint-and-2D-Motion based generative method for Face Video Compression, namely FVC-3K2M, which can well ensure perceptual compensation and visual consistency between motion description and face reconstruction. In particular, the temporal evolution of face video can be characterized into separate 3D keypoints from the global and local perspectives, entailing great coding flexibility and accurate motion representation. Moreover, a cascade motion conversion mechanism is further proposed to internally convert 3D keypoints to 2D dense motion, enforcing the face video reconstruction to be perceptually realistic. Finally, an adaptive reference frame selection scheme is developed to enhance the adaptation of various temporal movements. Experimental results show that the proposed scheme can realize reliable video communication in the extremely limited bandwidth, e.g., 2 kbps. Compared to the state-of-the-art video coding standards and the latest face video compression methods, extensive comparisons demonstrate that our proposed scheme achieves superior compression performance in terms of multiple quality evaluations.
Förderung
- “Leading Goose” Research and Development Program of Zhejiang Province
- National Natural Science Foundation of China
- Fundamental Research Funds for the Central Universities
Zitieren
APA
Wang, Z., Chen, B., Wang, S., Wang, S., Ye, Y., & Ma, S. (2024). Ultra-Low Bitrate Face Video Compression Based on Conversions From 3D Keypoints to 2D Motion Map. IEEE Transactions on Image Processing, 33, 6850-6864. https://doi.org/10.1109/tip.2024.3518100
Vancouver
Wang Z, Chen B, Wang S, Wang S, Ye Y, Ma S. Ultra-Low Bitrate Face Video Compression Based on Conversions From 3D Keypoints to 2D Motion Map. IEEE Transactions on Image Processing. 2024;33:6850-6864. doi:10.1109/tip.2024.3518100
BibTeX
@article{Wang2024,
author = {Wang, Zhao and Chen, Bolin and Wang, Shurun and Wang, Shiqi and Ye, Yan and Ma, Siwei},
title = {Ultra-Low Bitrate Face Video Compression Based on Conversions From 3D Keypoints to 2D Motion Map},
journal = {IEEE Transactions on Image Processing},
year = {2024},
volume = {33},
pages = {6850--6864},
doi = {10.1109/tip.2024.3518100}
}
EUVIMED