EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

A Study on Webtoon Generation Using CLIP and Diffusion Models

Kyungho Yu, Hyoungju Kim, Jeongin Kim, Chanjun Chun, Pankoo Kim

Electronics · 21.09.2023 · 12 (18) · 3983

DOI: 10.3390/electronics12183983 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Electronics, online erschienen am 21.09.2023. Verfasst von Kyungho Yu und 4 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 22 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Chosun University, ‘Technology Commercialization Collaboration Platform Construction’ project of the INNOPOLIS FOUNDATION.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This study focuses on harnessing deep-learning-based text-to-image transformation techniques to help webtoon creators’ creative outputs. We converted publicly available datasets (e.g., MSCOCO) into a multimodal webtoon dataset using CartoonGAN. First, the dataset was leveraged for training contrastive language image pre-training (CLIP), a model composed of multi-lingual BERT and a Vision Transformer that learnt to associate text with images. Second, a pre-trained diffusion model was employed to generate webtoons through text and text-similar image input. The webtoon dataset comprised treatments (i.e., textual descriptions) paired with their corresponding webtoon illustrations. CLIP (operating through contrastive learning) extracted features from different data modalities and aligned similar data more closely within the same feature space while pushing dissimilar data apart. This model learnt the relationships between various modalities in multimodal data. To generate webtoons using the diffusion model, the process involved providing the CLIP features of the desired webtoon’s text with those of the most text-similar image to a pre-trained diffusion model. Experiments were conducted using both single- and continuous-text inputs to generate webtoons. In the experiments, both single-text and continuous-text inputs were used to generate webtoons, and the results showed an inception score of 7.14 when using continuous-text inputs. The text-to-image technology developed here could streamline the webtoon creation process for artists by enabling the efficient generation of webtoons based on the provided text. However, the current inability to generate webtoons from multiple sentences or images while maintaining a consistent artistic style was noted. Therefore, further research is imperative to develop a text-to-image model capable of handling multi-sentence and -lingual input while ensuring coherence in the artistic style across the generated webtoon images.

Förderung

  • Chosun University
  • ‘Technology Commercialization Collaboration Platform Construction’ project of the INNOPOLIS FOUNDATION

Zitieren

APA

Yu, K., Kim, H., Kim, J., Chun, C., & Kim, P. (2023). A Study on Webtoon Generation Using CLIP and Diffusion Models. Electronics, 12(18), 3983. https://doi.org/10.3390/electronics12183983

Vancouver

Yu K, Kim H, Kim J, Chun C, Kim P. A Study on Webtoon Generation Using CLIP and Diffusion Models. Electronics. 2023;12(18):3983. doi:10.3390/electronics12183983

BibTeX
@article{Yu2023,
  author = {Yu, Kyungho and Kim, Hyoungju and Kim, Jeongin and Chun, Chanjun and Kim, Pankoo},
  title = {A Study on Webtoon Generation Using CLIP and Diffusion Models},
  journal = {Electronics},
  year = {2023},
  volume = {12},
  number = {18},
  pages = {3983},
  doi = {10.3390/electronics12183983}
}
Kostenlos starten →