EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

EmoStyle: Emotion-Aware Semantic Image Manipulation with Audio Guidance

Qiwei Shen, Junjie Xu, Jiahao Mei, Xingjiao Wu, Daoguo Dong

Applied Sciences · 10.04.2024 · 14 (8) · 3193

DOI: 10.3390/app14083193 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 10.04.2024. Verfasst von Qiwei Shen und 4 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 48 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

With the flourishing development of generative models, image manipulation is receiving increasing attention. Rather than text modality, several elegant designs have delved into leveraging audio to manipulate images. However, existing methodologies mainly focus on image generation conditional on semantic alignment, ignoring the vivid affective information depicted in the audio. We propose an Emotion-aware StyleGAN Manipulator (EmoStyle), a framework where affective information from audio can be explicitly extracted and further utilized during image manipulation. Specifically, we first leverage the multi-modality model ImageBind for initial cross-modal retrieval between images and music, and select the music-related image for further manipulation. Simultaneously, by extracting sentiment polarity from the lyrics of the audio, we generate an emotionally rich auxiliary music branch to accentuate the affective information. We then leverage pre-trained encoders to encode audio and the audio-related image into the same embedding space. With the aligned embeddings, we manipulate the image via a direct latent optimization method. We conduct objective and subjective evaluations on the generated images, and our results show that our framework is capable of generating images with specified human emotions conveyed in the audio.

Zitieren

APA

Shen, Q., Xu, J., Mei, J., Wu, X., & Dong, D. (2024). EmoStyle: Emotion-Aware Semantic Image Manipulation with Audio Guidance. Applied Sciences, 14(8), 3193. https://doi.org/10.3390/app14083193

Vancouver

Shen Q, Xu J, Mei J, Wu X, Dong D. EmoStyle: Emotion-Aware Semantic Image Manipulation with Audio Guidance. Applied Sciences. 2024;14(8):3193. doi:10.3390/app14083193

BibTeX
@article{Shen2024,
  author = {Shen, Qiwei and Xu, Junjie and Mei, Jiahao and Wu, Xingjiao and Dong, Daoguo},
  title = {EmoStyle: Emotion-Aware Semantic Image Manipulation with Audio Guidance},
  journal = {Applied Sciences},
  year = {2024},
  volume = {14},
  number = {8},
  pages = {3193},
  doi = {10.3390/app14083193}
}
Kostenlos starten →