Semantic-Aware Data Augmentation for Text-to-Image Synthesis
Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (6) · 5098-5107
DOI: 10.1609/aaai.v38i6.28315 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Zhaorui Tan und 2 weiteren Autor:innen. Die Arbeit wurde bisher 7-mal zitiert (Crossref).
Abstract (Originalsprache)
Data augmentation has been recently leveraged as an effective regularizer in various vision-language deep neural networks. However, in text-to-image synthesis (T2Isyn), current augmentation wisdom still suffers from the semantic mismatch between augmented paired data. Even worse, semantic collapse may occur when generated images are less semantically constrained. In this paper, we develop a novel Semantic-aware Data Augmentation (SADA) framework dedicated to T2Isyn. In particular, we propose to augment texts in the semantic space via an Implicit Textual Semantic Preserving Augmentation, in conjunction with a specifically designed Image Semantic Regularization Loss as Generated Image Semantic Conservation, to cope well with semantic mismatch and collapse. As one major contribution, we theoretically show that Implicit Textual Semantic Preserving Augmentation can certify better text-image consistency while Image Semantic Regularization Loss regularizing the semantics of generated images would avoid semantic collapse and enhance image quality. Extensive experiments validate that SADA enhances text-image consistency and improves image quality significantly in T2Isyn models across various backbones. Especially, incorporating SADA during the tuning process of Stable Diffusion models also yields performance improvements.
Zitieren
APA
Tan, Z., Yang, X., & Huang, K. (2024). Semantic-Aware Data Augmentation for Text-to-Image Synthesis. Proceedings of the AAAI Conference on Artificial Intelligence, 38(6), 5098-5107. https://doi.org/10.1609/aaai.v38i6.28315
Vancouver
Tan Z, Yang X, Huang K. Semantic-Aware Data Augmentation for Text-to-Image Synthesis. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(6):5098-5107. doi:10.1609/aaai.v38i6.28315
BibTeX
@article{Tan2024,
author = {Tan, Zhaorui and Yang, Xi and Huang, Kaizhu},
title = {Semantic-Aware Data Augmentation for Text-to-Image Synthesis},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2024},
volume = {38},
number = {6},
pages = {5098--5107},
doi = {10.1609/aaai.v38i6.28315}
}
EUVIMED