EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Sound2Vision: Generating Diverse Visuals from Audio through Cross-Modal Latent Alignment

Sung-Bin Kim, Arda Senocak, Hyunwoo Ha, Tae-Hyun Oh

International Journal of Computer Vision · 2026-12 · 134 (12)

DOI: 10.1007/s11263-026-03106-5 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Computer Vision, online erschienen am 21.11.2026. Verfasst von Sung-Bin Kim und 3 weiteren Autor:innen. Die Arbeit nennt 99 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Ministry of Science and ICT.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

How does audio describe the world around us? In this work, we propose a method for generating images of visual scenes from diverse in-the-wild sounds. This cross-modal generation task is challenging due to the significant information gap between auditory and visual signals. We address this challenge by designing a model that aligns audio-visual modalities by enriching audio features with visual information and translating them into the visual latent space. These features are then fed into the pre-trained image generator to produce images. To enhance image quality, we use sound source localization to select audio-visual pairs with strong cross-modal correlations. Our method achieves substantially better results on the VEGAS and VGGSound datasets compared to previous work and demonstrates control over the generation process through simple manipulations to the input waveform or latent space. Furthermore, we analyze the geometric properties of the learned embedding space and demonstrate that our learning approach effectively aligns audio-visual signals for cross-modal generation. Based on this analysis, we show that our method is agnostic to specific design choices, showing its generalizability by integrating various model architectures and different types of audio-visual data.

Förderung

  • Ministry of Science and ICT

Zitieren

APA

Kim, S., Senocak, A., Ha, H., & Oh, T. (2026). Sound2Vision: Generating Diverse Visuals from Audio through Cross-Modal Latent Alignment. International Journal of Computer Vision, 134(12). https://doi.org/10.1007/s11263-026-03106-5

Vancouver

Kim S, Senocak A, Ha H, Oh T. Sound2Vision: Generating Diverse Visuals from Audio through Cross-Modal Latent Alignment. International Journal of Computer Vision. 2026;134(12). doi:10.1007/s11263-026-03106-5

BibTeX
@article{Kim2026,
  author = {Kim, Sung-Bin and Senocak, Arda and Ha, Hyunwoo and Oh, Tae-Hyun},
  title = {Sound2Vision: Generating Diverse Visuals from Audio through Cross-Modal Latent Alignment},
  journal = {International Journal of Computer Vision},
  year = {2026},
  volume = {134},
  number = {12},
  doi = {10.1007/s11263-026-03106-5}
}
Kostenlos starten →