Create Your World: Lifelong Text-to-Image Diffusion
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-09 · 46 (9) · 6454-6470
DOI: 10.1109/tpami.2024.3382753 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-09. Verfasst von Gan Sun und 5 weiteren Autor:innen. Die Arbeit wurde bisher 54-mal zitiert (Crossref) und nennt 44 Literaturangaben. Gefördert von: National Natural Science Foundation of China, CAS-Youth Innovation Promotion Association Scholarship.
Abstract (Originalsprache)
Text-to-image generative models can produce diverse high-quality images of concepts with a text prompt, which have demonstrated excellent ability in image generation, image translation, etc. We in this work study the problem of synthesizing instantiations of a user's own concepts in a never-ending manner, i.e., create your world, where the new concepts from user are quickly learned with a few examples. To achieve this goal, we propose a L ife l ong text-to-image D iffusion M odel (L $^{2}$ 2 DM), which intends to overcome knowledge “catastrophic forgetting” for the past encountered concepts, and semantic “catastrophic neglecting” for one or more concepts in the text prompt. In respect of knowledge “catastrophic forgetting”, our L $^{2}$ 2 DM framework devises a task-aware memory enhancement module and an elastic-concept distillation module, which could respectively safeguard the knowledge of both prior concepts and each past personalized concept. When generating images with a user text prompt, the solution to semantic “catastrophic neglecting” is that a concept attention artist module can alleviate the semantic neglecting from concept aspect, and an orthogonal attention module can reduce the semantic binding from attribute aspect. To the end, our model can generate more faithful image across a range of continual text prompts in terms of both qualitative and quantitative metrics, when comparing with the related state-of-the-art models.
Förderung
- National Natural Science Foundation of China
- CAS-Youth Innovation Promotion Association Scholarship
Zitieren
APA
Sun, G., Liang, W., Dong, J., Li, J., Ding, Z., & Cong, Y. (2024). Create Your World: Lifelong Text-to-Image Diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(9), 6454-6470. https://doi.org/10.1109/tpami.2024.3382753
Vancouver
Sun G, Liang W, Dong J, Li J, Ding Z, Cong Y. Create Your World: Lifelong Text-to-Image Diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(9):6454-6470. doi:10.1109/tpami.2024.3382753
BibTeX
@article{Sun2024,
author = {Sun, Gan and Liang, Wenqi and Dong, Jiahua and Li, Jun and Ding, Zhengming and Cong, Yang},
title = {Create Your World: Lifelong Text-to-Image Diffusion},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2024},
volume = {46},
number = {9},
pages = {6454--6470},
doi = {10.1109/tpami.2024.3382753}
}
EUVIMED