SGEdit: Bridging LLM with Text2Image Generative Model for Scene Graph-based Image Editing
ACM Transactions on Graphics · 19.12.2024 · 43 (6) · 1-16
DOI: 10.1145/3687957 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Graphics, online erschienen am 19.11.2024. Verfasst von Zhiyuan Zhang und 2 weiteren Autor:innen. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 59 Literaturangaben. Gefördert von: Hong Kong Research Grants Council (RGC) GRF Scheme.
Abstract (Originalsprache)
Scene graphs offer a structured, hierarchical representation of images, with nodes and edges symbolizing objects and the relationships among them. It can serve as a natural interface for image editing, dramatically improving precision and flexibility. Leveraging this benefit, we introduce a new framework that integrates large language model (LLM) with Text2Image generative model for scene graph-based image editing. This integration enables precise modifications at the object level and creative recomposition of scenes without compromising overall image integrity. Our approach involves two primary stages: 1) Utilizing a LLM-driven scene parser, we construct an image's scene graph, capturing key objects and their interrelationships, as well as parsing fine-grained attributes such as object masks and descriptions. These annotations facilitate concept learning with a fine-tuned diffusion model, representing each object with an optimized token and detailed description prompt. 2) During the image editing phase, a LLM editing controller guides the edits towards specific areas. These edits are then implemented by an attention-modulated diffusion editor, utilizing the fine-tuned model to perform object additions, deletions, replacements, and adjustments. Through extensive experiments, we demonstrate that our framework significantly outperforms existing image editing methods in terms of editing precision and scene aesthetics. Our code is available at https://bestzzhang.github.io/SGEdit.
Förderung
- Hong Kong Research Grants Council (RGC) GRF Scheme
Zitieren
APA
Zhang, Z., Chen, D., & Liao, J. (2024). SGEdit: Bridging LLM with Text2Image Generative Model for Scene Graph-based Image Editing. ACM Transactions on Graphics, 43(6), 1-16. https://doi.org/10.1145/3687957
Vancouver
Zhang Z, Chen D, Liao J. SGEdit: Bridging LLM with Text2Image Generative Model for Scene Graph-based Image Editing. ACM Transactions on Graphics. 2024;43(6):1-16. doi:10.1145/3687957
BibTeX
@article{Zhang2024,
author = {Zhang, Zhiyuan and Chen, DongDong and Liao, Jing},
title = {SGEdit: Bridging LLM with Text2Image Generative Model for Scene Graph-based Image Editing},
journal = {ACM Transactions on Graphics},
year = {2024},
volume = {43},
number = {6},
pages = {1--16},
doi = {10.1145/3687957}
}
EUVIMED