Personalization as a Shortcut for Few-Shot Backdoor Attack against Text-to-Image Diffusion Models
Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (19) · 21169-21178
DOI: 10.1609/aaai.v38i19.30110 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Yihao Huang und 8 weiteren Autor:innen. Die Arbeit wurde bisher 31-mal zitiert (Crossref).
Abstract (Originalsprache)
Although recent personalization methods have democratized high-resolution image synthesis by enabling swift concept acquisition with minimal examples and lightweight computation, they also present an exploitable avenue for highly accessible backdoor attacks. This paper investigates a critical and unexplored aspect of text-to-image (T2I) diffusion models - their potential vulnerability to backdoor attacks via personalization. By studying the prompt processing of popular personalization methods (epitomized by Textual Inversion and DreamBooth), we have devised dedicated personalization-based backdoor attacks according to the different ways of dealing with unseen tokens and divide them into two families: nouveau-token and legacy-token backdoor attacks. In comparison to conventional backdoor attacks involving the fine-tuning of the entire text-to-image diffusion model, our proposed personalization-based backdoor attack method can facilitate more tailored, efficient, and few-shot attacks. Through comprehensive empirical study, we endorse the utilization of the nouveau-token backdoor attack due to its impressive effectiveness, stealthiness, and integrity, markedly outperforming the legacy-token backdoor attack.
Zitieren
APA
Huang, Y., Juefei-Xu, F., Guo, Q., Zhang, J., Wu, Y., Hu, M., Li, T., Pu, G., & Liu, Y. (2024). Personalization as a Shortcut for Few-Shot Backdoor Attack against Text-to-Image Diffusion Models. Proceedings of the AAAI Conference on Artificial Intelligence, 38(19), 21169-21178. https://doi.org/10.1609/aaai.v38i19.30110
Vancouver
Huang Y, Juefei-Xu F, Guo Q, Zhang J, Wu Y, Hu M, et al. Personalization as a Shortcut for Few-Shot Backdoor Attack against Text-to-Image Diffusion Models. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(19):21169-21178. doi:10.1609/aaai.v38i19.30110
BibTeX
@article{Huang2024,
author = {Huang, Yihao and Juefei-Xu, Felix and Guo, Qing and Zhang, Jie and Wu, Yutong and Hu, Ming and Li, Tianlin and Pu, Geguang and Liu, Yang},
title = {Personalization as a Shortcut for Few-Shot Backdoor Attack against Text-to-Image Diffusion Models},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2024},
volume = {38},
number = {19},
pages = {21169--21178},
doi = {10.1609/aaai.v38i19.30110}
}
EUVIMED