EUVIMEDEUVIMED entdecken
Crossref

Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models

Rinon Gal, Moab Arar, Yuval Atzmon, Amit H. Bermano, Gal Chechik, Daniel Cohen-Or

ACM Transactions on Graphics · 2023-08 · 42 (4) · 1-13

DOI: 10.1145/3592133 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Graphics, online erschienen am 26.07.2023. Verfasst von Rinon Gal und 5 weiteren Autor:innen. Die Arbeit wurde bisher 177-mal zitiert (Crossref) und nennt 85 Literaturangaben. Gefördert von: BSF, ISF.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Text-to-image personalization aims to teach a pre-trained diffusion model to reason about novel, user provided concepts, embedding them into new scenes guided by natural language prompts. However, current personalization approaches struggle with lengthy training times, high storage requirements or loss of identity. To overcome these limitations, we propose an encoder-based domain-tuning approach. Our key insight is that by underfitting on a large set of concepts from a given domain, we can improve generalization and create a model that is more amenable to quickly adding novel concepts from the same domain. Specifically, we employ two components: First, an encoder that takes as an input a single image of a target concept from a given domain, e.g. a specific face, and learns to map it into a word-embedding representing the concept. Second, a set of regularized weight-offsets for the text-to-image model that learn how to effectively injest additional concepts. Together, these components are used to guide the learning of unseen concepts, allowing us to personalize a model using only a single image and as few as 5 training steps --- accelerating personalization from dozens of minutes to seconds , while preserving quality. Code and trained encoders will be available at our project page.

Förderung

  • BSF
  • ISF

Zitieren

APA

Gal, R., Arar, M., Atzmon, Y., Bermano, A. H., Chechik, G., & Cohen-Or, D. (2023). Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models. ACM Transactions on Graphics, 42(4), 1-13. https://doi.org/10.1145/3592133

Vancouver

Gal R, Arar M, Atzmon Y, Bermano AH, Chechik G, Cohen-Or D. Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models. ACM Transactions on Graphics. 2023;42(4):1-13. doi:10.1145/3592133

BibTeX
@article{Gal2023,
  author = {Gal, Rinon and Arar, Moab and Atzmon, Yuval and Bermano, Amit H. and Chechik, Gal and Cohen-Or, Daniel},
  title = {Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models},
  journal = {ACM Transactions on Graphics},
  year = {2023},
  volume = {42},
  number = {4},
  pages = {1--13},
  doi = {10.1145/3592133}
}
Kostenlos starten →