EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Towards Scene-Aware Video-to-Spatial Audio Generation

Jaeyeon Kim, Heeseung Yun, Gunhee Kim

International Journal of Computer Vision · 2026-04 · 134 (4)

DOI: 10.1007/s11263-025-02610-4 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Computer Vision, online erschienen am 09.03.2026. Verfasst von Jaeyeon Kim und 2 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 100 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: IITP.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Spatial audio is essential for enhancing the immersiveness of audio-visual experiences, yet its production typically demands complex recording systems and specialized expertise. In this work, we address a novel problem of generating first-order ambisonics, a widely used spatial audio format, directly from silent videos. To support this task, we develop comprehensive evaluation metrics that capture both standard video-to-audio generation quality and spatial coherence among multiple channels. We introduce YT-Ambigen, a dataset comprising 102K YouTube video clips paired with first-order ambisonics tailored for audio generation, and its expanded version YT-Ambigen+ containing 3x more clips with a rigorously validated high-quality test subset of 19.3K clips. Furthermore, we present Video-to-Spatial Audio Generation (ViSAGe), an end-to-end framework that generates first-order ambisonics from silent videos by leveraging CLIP features, patchwise energy maps, and neural audio codecs with rotation augmentation. To address efficiency challenges, we propose a variant coined ViSAGe-SC (Single Codebook), which replaces complex residual codebooks with an optimized single codebook approach, achieving 4x faster training and 5x faster inference while maintaining superior performance. ViSAGe-SC incorporates heterogeneous codec chaining for postprocessing and candidate reranking for inference-time refinement. Experimental results demonstrate that our approach outperforms several V2A models across spatial metrics and displays competitive performance in semantic quality, generating high-quality spatial audio from video input.

Förderung

  • IITP

Zitieren

APA

Kim, J., Yun, H., & Kim, G. (2026). Towards Scene-Aware Video-to-Spatial Audio Generation. International Journal of Computer Vision, 134(4). https://doi.org/10.1007/s11263-025-02610-4

Vancouver

Kim J, Yun H, Kim G. Towards Scene-Aware Video-to-Spatial Audio Generation. International Journal of Computer Vision. 2026;134(4). doi:10.1007/s11263-025-02610-4

BibTeX
@article{Kim2026,
  author = {Kim, Jaeyeon and Yun, Heeseung and Kim, Gunhee},
  title = {Towards Scene-Aware Video-to-Spatial Audio Generation},
  journal = {International Journal of Computer Vision},
  year = {2026},
  volume = {134},
  number = {4},
  doi = {10.1007/s11263-025-02610-4}
}
Kostenlos starten →