EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Video Object Segmentation-Aware Audio Generation

Ilpo Viertola, Vladimir Iashin, Esa Rahtu

International Journal of Computer Vision · 2026-07 · 134 (7)

DOI: 10.1007/s11263-026-02911-2 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Computer Vision, online erschienen am 23.06.2026. Verfasst von Ilpo Viertola und 2 weiteren Autor:innen. Die Arbeit nennt 78 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Academy of Finland.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Existing multimodal audio generation models often lack precise user control, which limits their applicability in professional Foley workflows. In particular, these models focus on the entire video and do not provide precise methods for prioritizing a specific object within a scene, generating unnecessary background sounds, or focusing on the wrong objects. To address this gap, we introduce the novel task of video object segmentation-aware audio generation, which explicitly conditions sound synthesis on object-level segmentation maps. We present SAGANet, a new multimodal generative model that enables controllable audio generation for musical instruments by leveraging visual segmentation masks along with video and textual cues. Our model provides users with fine-grained and visually localized control over audio generation. To support this task and further research on segmentation-aware Foley, we propose Segmented Music Solos, a benchmark dataset of musical instrument performance videos with segmentation information. Our method demonstrates substantial improvements over current state-of-the-art methods and sets a new standard for controllable, high-fidelity Foley synthesis for musical audio. Code, samples, and Segmented Music Solos are available at http://saganet.notion.site .

Förderung

  • Academy of Finland

Zitieren

APA

Viertola, I., Iashin, V., & Rahtu, E. (2026). Video Object Segmentation-Aware Audio Generation. International Journal of Computer Vision, 134(7). https://doi.org/10.1007/s11263-026-02911-2

Vancouver

Viertola I, Iashin V, Rahtu E. Video Object Segmentation-Aware Audio Generation. International Journal of Computer Vision. 2026;134(7). doi:10.1007/s11263-026-02911-2

BibTeX
@article{Viertola2026,
  author = {Viertola, Ilpo and Iashin, Vladimir and Rahtu, Esa},
  title = {Video Object Segmentation-Aware Audio Generation},
  journal = {International Journal of Computer Vision},
  year = {2026},
  volume = {134},
  number = {7},
  doi = {10.1007/s11263-026-02911-2}
}
Kostenlos starten →