TIP-Editor: An Accurate 3D Editor Following Both Text-Prompts And Image-Prompts
ACM Transactions on Graphics · 19.07.2024 · 43 (4) · 1-12
DOI: 10.1145/3658205 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Graphics, online erschienen am 19.07.2024. Verfasst von Jingyu Zhuang und 5 weiteren Autor:innen. Die Arbeit wurde bisher 78-mal zitiert (Crossref) und nennt 61 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Open Project Program of State Key Laboratory of Virtual Reality Technology and Systems, Beihang University.
Abstract (Originalsprache)
Text-driven 3D scene editing has gained significant attention owing to its convenience and user-friendliness. However, existing methods still lack accurate control of the specified appearance and location of the editing result due to the inherent limitations of the text description. To this end, we propose a 3D scene editing framework, TIP-Editor, that accepts both text and image prompts and a 3D bounding box to specify the editing region. With the image prompt, users can conveniently specify the detailed appearance/style of the target content in complement to the text description, enabling accurate control of the appearance. Specifically, TIP-Editor employs a stepwise 2D personalization strategy to better learn the representation of the existing scene and the reference image, in which a localization loss is proposed to encourage correct object placement as specified by the bounding box. Additionally, TIP-Editor utilizes explicit and flexible 3D Gaussian splatting (GS) as the 3D representation to facilitate local editing while keeping the background unchanged. Extensive experiments have demonstrated that TIP-Editor conducts accurate editing following the text and image prompts in the specified bounding box region, consistently outperforming the baselines in editing quality, and the alignment to the prompts, qualitatively and quantitatively.
Förderung
- National Natural Science Foundation of China
- Open Project Program of State Key Laboratory of Virtual Reality Technology and Systems, Beihang University
Zitieren
APA
Zhuang, J., Kang, D., Cao, Y., Li, G., Lin, L., & Shan, Y. (2024). TIP-Editor: An Accurate 3D Editor Following Both Text-Prompts And Image-Prompts. ACM Transactions on Graphics, 43(4), 1-12. https://doi.org/10.1145/3658205
Vancouver
Zhuang J, Kang D, Cao Y, Li G, Lin L, Shan Y. TIP-Editor: An Accurate 3D Editor Following Both Text-Prompts And Image-Prompts. ACM Transactions on Graphics. 2024;43(4):1-12. doi:10.1145/3658205
BibTeX
@article{Zhuang2024,
author = {Zhuang, Jingyu and Kang, Di and Cao, Yan-Pei and Li, Guanbin and Lin, Liang and Shan, Ying},
title = {TIP-Editor: An Accurate 3D Editor Following Both Text-Prompts And Image-Prompts},
journal = {ACM Transactions on Graphics},
year = {2024},
volume = {43},
number = {4},
pages = {1--12},
doi = {10.1145/3658205}
}
EUVIMED