EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models

Yuxin Zhang, Weiming Dong, Fan Tang, Nisha Huang, Haibin Huang, Chongyang Ma, Tong-Yee Lee, Oliver Deussen, Changsheng Xu

ACM Transactions on Graphics · 05.12.2023 · 42 (6) · 1-14

DOI: 10.1145/3618342 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Graphics, online erschienen am 05.12.2023. Verfasst von Yuxin Zhang und 8 weiteren Autor:innen. Die Arbeit wurde bisher 107-mal zitiert (Crossref) und nennt 72 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, Deutsche Forschungsgemeinschaft, Beijing Natural Science Foundation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Personalizing generative models offers a way to guide image generation with user-provided references. Current personalization methods can invert an object or concept into the textual conditioning space and compose new natural sentences for text-to-image diffusion models. However, representing and editing specific visual attributes such as material, style, and layout remains a challenge, leading to a lack of disentanglement and editability. To address this problem, we propose a novel approach that leverages the step-by-step generation process of diffusion models, which generate images from low to high frequency information, providing a new perspective on representing, generating, and editing images. We develop the Prompt Spectrum Space P*, an expanded textual conditioning space, and a new image representation method called ProSpect. ProSpect represents an image as a collection of inverted textual token embeddings encoded from per-stage prompts, where each prompt corresponds to a specific generation stage (i.e., a group of consecutive steps) of the diffusion model. Experimental results demonstrate that P* and ProSpect offer better disentanglement and controllability compared to existing methods. We apply ProSpect in various personalized attribute-aware image generation applications, such as image-guided or text-driven manipulations of materials, style, and layout, achieving previously unattainable results from a single image input without fine-tuning the diffusion models. Our source code is available at https://github.com/zyxElsa/ProSpect.

Förderung

  • National Natural Science Foundation of China
  • Deutsche Forschungsgemeinschaft
  • Beijing Natural Science Foundation
  • National Key R&D Program of China
  • National Science and Technology Council

Zitieren

APA

Zhang, Y., Dong, W., Tang, F., Huang, N., Huang, H., Ma, C., Lee, T., Deussen, O., & Xu, C. (2023). ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models. ACM Transactions on Graphics, 42(6), 1-14. https://doi.org/10.1145/3618342

Vancouver

Zhang Y, Dong W, Tang F, Huang N, Huang H, Ma C, et al. ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models. ACM Transactions on Graphics. 2023;42(6):1-14. doi:10.1145/3618342

BibTeX
@article{Zhang2023,
  author = {Zhang, Yuxin and Dong, Weiming and Tang, Fan and Huang, Nisha and Huang, Haibin and Ma, Chongyang and Lee, Tong-Yee and Deussen, Oliver and Xu, Changsheng},
  title = {ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models},
  journal = {ACM Transactions on Graphics},
  year = {2023},
  volume = {42},
  number = {6},
  pages = {1--14},
  doi = {10.1145/3618342}
}
Kostenlos starten →