Real-Time Radiance Fields for Single-Image Portrait View Synthesis
ACM Transactions on Graphics · 2023-08 · 42 (4) · 1-15
DOI: 10.1145/3592460 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Graphics, online erschienen am 26.07.2023. Verfasst von Alex Trevithick und 9 weiteren Autor:innen. Die Arbeit wurde bisher 73-mal zitiert (Crossref) und nennt 102 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: ONR, NSF, Defense Advanced Research Projects Agency.
Abstract (Originalsprache)
We present a one-shot method to infer and render a photorealistic 3D representation from a single unposed image (e.g., face portrait) in real-time. Given a single RGB input, our image encoder directly predicts a canonical triplane representation of a neural radiance field for 3D-aware novel view synthesis via volume rendering. Our method is fast (24 fps) on consumer hardware, and produces higher quality results than strong GAN-inversion baselines that require test-time optimization. To train our triplane encoder pipeline, we use only synthetic data, showing how to distill the knowledge from a pretrained 3D GAN into a feedforward encoder. Technical contributions include a Vision Transformer-based triplane encoder, a camera data augmentation strategy, and a well-designed loss function for synthetic data training. We benchmark against the state-of-the-art methods, demonstrating significant improvements in robustness and image quality in challenging real-world settings. We showcase our results on portraits of faces (FFHQ) and cats (AFHQ), but our algorithm can also be applied in the future to other categories with a 3D-aware image generator.
Förderung
- ONR
- NSF
- Defense Advanced Research Projects Agency
Zitieren
APA
Trevithick, A., Chan, M., Stengel, M., Chan, E., Liu, C., Yu, Z., Khamis, S., Chandraker, M., Ramamoorthi, R., & Nagano, K. (2023). Real-Time Radiance Fields for Single-Image Portrait View Synthesis. ACM Transactions on Graphics, 42(4), 1-15. https://doi.org/10.1145/3592460
Vancouver
Trevithick A, Chan M, Stengel M, Chan E, Liu C, Yu Z, et al. Real-Time Radiance Fields for Single-Image Portrait View Synthesis. ACM Transactions on Graphics. 2023;42(4):1-15. doi:10.1145/3592460
BibTeX
@article{Trevithick2023,
author = {Trevithick, Alex and Chan, Matthew and Stengel, Michael and Chan, Eric and Liu, Chao and Yu, Zhiding and Khamis, Sameh and Chandraker, Manmohan and Ramamoorthi, Ravi and Nagano, Koki},
title = {Real-Time Radiance Fields for Single-Image Portrait View Synthesis},
journal = {ACM Transactions on Graphics},
year = {2023},
volume = {42},
number = {4},
pages = {1--15},
doi = {10.1145/3592460}
}
EUVIMED