CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image
ACM Transactions on Graphics · 01.08.2025 · 44 (4) · 1-19
DOI: 10.1145/3730841 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Graphics, online erschienen am 27.07.2025. Verfasst von Kaixin Yao und 8 weiteren Autor:innen. Die Arbeit wurde bisher 45-mal zitiert (Crossref) und nennt 97 Literaturangaben. Gefördert von: National Key R&D Program of China, NSFC programs, STCSM.
Abstract (Originalsprache)
Recovering high-quality 3D scenes from a single RGB image is a challenging task in computer graphics. Current methods often struggle with domain-specific limitations or low-quality object generation. To address these, we propose CAST (Component-Aligned 3D Scene Reconstruction from a Single RGB Image), a novel method for 3D scene reconstruction. CAST starts by extracting object-level 2D segmentation and relative depth information from the input image, followed by using a GPT-based model to analyze inter-object spatial relations. This enables understanding of how objects relate to each other within the scene, ensuring more coherent reconstruction. CAST then employs an occlusion-aware large-scale 3D generation model to independently generate each object's full geometry, using Masked Auto Encoder (MAE) and point cloud conditioning to mitigate the effects of occlusions and partial object information, ensuring accurate alignment with the source image's geometry and texture. To align each object with the scene, the alignment generation model computes the necessary transformations, allowing the generated meshes to be accurately placed and integrated into the scene's point cloud. Finally, CAST applies a physics-aware correction mechanism, which leverages a fine-grained relation graph to generate a constraint graph. This graph guides the optimization of object poses, ensuring physical consistency and spatial coherence. By utilizing Signed Distance Fields (SDF), the model effectively addresses issues such as occlusions, object penetration, and floating objects, ensuring that the generated scene accurately reflects real-world physical interactions. Experimental results demonstrate that CAST significantly improves the quality of single-image 3D scene reconstruction, offering enhanced realism and accuracy in scene understanding and reconstruction tasks. CAST has practical applications in virtual content creation, such as immersive game environments and film production, where real-world setups can be seamlessly integrated into virtual landscapes. Additionally, CAST can be leveraged in robotics, enabling efficient real-to-simulation workflows and providing realistic, scalable simulation environments for robotic systems.
Förderung
- National Key R&D Program of China
- NSFC programs
- STCSM
- SHMEC
Zitieren
APA
Yao, K., Zhang, L., Yan, X., Zeng, Y., Zhang, Q., Xu, L., Yang, W., Gu, J., & Yu, J. (2025). CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image. ACM Transactions on Graphics, 44(4), 1-19. https://doi.org/10.1145/3730841
Vancouver
Yao K, Zhang L, Yan X, Zeng Y, Zhang Q, Xu L, et al. CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image. ACM Transactions on Graphics. 2025;44(4):1-19. doi:10.1145/3730841
BibTeX
@article{Yao2025,
author = {Yao, Kaixin and Zhang, Longwen and Yan, Xinhao and Zeng, Yan and Zhang, Qixuan and Xu, Lan and Yang, Wei and Gu, Jiayuan and Yu, Jingyi},
title = {CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image},
journal = {ACM Transactions on Graphics},
year = {2025},
volume = {44},
number = {4},
pages = {1--19},
doi = {10.1145/3730841}
}
EUVIMED