Robust Dual Gaussian Splatting for Immersive Human-centric Volumetric Videos
ACM Transactions on Graphics · 19.12.2024 · 43 (6) · 1-15
DOI: 10.1145/3687926 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Graphics, online erschienen am 19.11.2024. Verfasst von Yuheng Jiang und 7 weiteren Autor:innen. Die Arbeit wurde bisher 28-mal zitiert (Crossref) und nennt 106 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-SA 4.0). Gefördert von: National Key R&D Program of China, NSFC programs, STCSM.
Abstract (Originalsprache)
Volumetric video represents a transformative advancement in visual media, enabling users to freely navigate immersive virtual experiences and narrowing the gap between digital and real worlds. However, the need for extensive manual intervention to stabilize mesh sequences and the generation of excessively large assets in existing workflows impedes broader adoption. In this paper, we present a novel Gaussian-based approach, dubbed DualGS , for real-time and high-fidelity playback of complex human performance with excellent compression ratios. Our key idea in DualGS is to separately represent motion and appearance using the corresponding skin and joint Gaussians. Such an explicit disentanglement can significantly reduce motion redundancy and enhance temporal coherence. We begin by initializing the DualGS and anchoring skin Gaussians to joint Gaussians at the first frame. Subsequently, we employ a coarse-to-fine training strategy for frame-by-frame human performance modeling. It includes a coarse alignment phase for overall motion prediction as well as a fine-grained optimization for robust tracking and high-fidelity rendering. To integrate volumetric video seamlessly into VR environments, we efficiently compress motion using entropy encoding and appearance using codec compression coupled with a persistent codebook. Our approach achieves a compression ratio of up to 120 times, only requiring approximately 350KB of storage per frame. We demonstrate the efficacy of our representation through photo-realistic, free-view experiences on VR headsets, enabling users to immersively watch musicians in performance and feel the rhythm of the notes at the performers' fingertips. Project page: https://nowheretrix.github.io/DualGS/.
Förderung
- National Key R&D Program of China
- NSFC programs
- STCSM
- SHMEC
Zitieren
APA
Jiang, Y., Shen, Z., Hong, Y., Guo, C., Wu, Y., Zhang, Y., Yu, J., & Xu, L. (2024). Robust Dual Gaussian Splatting for Immersive Human-centric Volumetric Videos. ACM Transactions on Graphics, 43(6), 1-15. https://doi.org/10.1145/3687926
Vancouver
Jiang Y, Shen Z, Hong Y, Guo C, Wu Y, Zhang Y, et al. Robust Dual Gaussian Splatting for Immersive Human-centric Volumetric Videos. ACM Transactions on Graphics. 2024;43(6):1-15. doi:10.1145/3687926
BibTeX
@article{Jiang2024,
author = {Jiang, Yuheng and Shen, Zhehao and Hong, Yu and Guo, Chengcheng and Wu, Yize and Zhang, Yingliang and Yu, Jingyi and Xu, Lan},
title = {Robust Dual Gaussian Splatting for Immersive Human-centric Volumetric Videos},
journal = {ACM Transactions on Graphics},
year = {2024},
volume = {43},
number = {6},
pages = {1--15},
doi = {10.1145/3687926}
}
EUVIMED