Quark: Real-time, High-resolution, and General Neural View Synthesis
ACM Transactions on Graphics · 19.12.2024 · 43 (6) · 1-20
DOI: 10.1145/3687953 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Graphics, online erschienen am 19.11.2024. Verfasst von John Flynn und 14 weiteren Autor:innen. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 98 Literaturangaben.
Abstract (Originalsprache)
We present a novel neural algorithm for performing high-quality, highresolution, real-time novel view synthesis. From a sparse set of input RGB images or videos streams, our network both reconstructs the 3D scene and renders novel views at 1080p resolution at 30fps on an NVIDIA A100. Our feed-forward network generalizes across a wide variety of datasets and scenes and produces state-of-the-art quality for a real-time method. Our quality approaches, and in some cases surpasses, the quality of some of the top offline methods. In order to achieve these results we use a novel combination of several key concepts, and tie them together into a cohesive and effective algorithm. We build on previous works that represent the scene using semi-transparent layers and use an iterative learned render-and-refine approach to improve those layers. Instead of flat layers, our method reconstructs layered depth maps (LDMs) that efficiently represent scenes with complex depth and occlusions. The iterative update steps are embedded in a multi-scale, UNet-style architecture to perform as much compute as possible at reduced resolution. Within each update step, to better aggregate the information from multiple input views, we use a specialized Transformer-based network component. This allows the majority of the per-input image processing to be performed in the input image space, as opposed to layer space, further increasing efficiency. Finally, due to the real-time nature of our reconstruction and rendering, we dynamically create and discard the internal 3D geometry for each frame, generating the LDM for each view. Taken together, this produces a novel and effective algorithm for view synthesis. Through extensive evaluation, we demonstrate that we achieve state-of-the-art quality at real-time rates.
Zitieren
APA
Flynn, J., Broxton, M., Murmann, L., Chai, L., DuVall, M., Godard, C., Heal, K., Kaza, S., Lombardi, S., Luo, X., Achar, S., Prabhu, K., Sun, T., Tsai, L., & Overbeck, R. (2024). Quark: Real-time, High-resolution, and General Neural View Synthesis. ACM Transactions on Graphics, 43(6), 1-20. https://doi.org/10.1145/3687953
Vancouver
Flynn J, Broxton M, Murmann L, Chai L, DuVall M, Godard C, et al. Quark: Real-time, High-resolution, and General Neural View Synthesis. ACM Transactions on Graphics. 2024;43(6):1-20. doi:10.1145/3687953
BibTeX
@article{Flynn2024,
author = {Flynn, John and Broxton, Michael and Murmann, Lukas and Chai, Lucy and DuVall, Matthew and Godard, Clément and Heal, Kathryn and Kaza, Srinivas and Lombardi, Stephen and Luo, Xuan and Achar, Supreeth and Prabhu, Kira and Sun, Tiancheng and Tsai, Lynn and Overbeck, Ryan},
title = {Quark: Real-time, High-resolution, and General Neural View Synthesis},
journal = {ACM Transactions on Graphics},
year = {2024},
volume = {43},
number = {6},
pages = {1--20},
doi = {10.1145/3687953}
}
EUVIMED