EUVIMEDEUVIMED entdecken
Crossref

Audio-Visual Segmentation with Semantics

Jinxing Zhou, Xuyang Shen, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, Yiran Zhong

International Journal of Computer Vision · 2025-04 · 133 (4) · 1644-1664

DOI: 10.1007/s11263-024-02261-x ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Computer Vision, online erschienen am 15.10.2024. Verfasst von Jinxing Zhou und 10 weiteren Autor:innen. Die Arbeit wurde bisher 69-mal zitiert (Crossref) und nennt 85 Literaturangaben. Gefördert von: Major Project of Anhui Province, National Key R&D Program of China, National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We propose a new problem called audio-visual segmentation (AVS), in which the goal is to output a pixel-level map of the object(s) that produce sound at the time of the image frame. To facilitate this research, we construct the first audio-visual segmentation benchmark, i.e., AVSBench, providing pixel-wise annotations for sounding objects in audible videos. It contains three subsets: AVSBench-object (Single-source subset, Multi-sources subset) and AVSBench-semantic (Semantic-labels subset). Accordingly, three settings are studied: 1) semi-supervised audio-visual segmentation with a single sound source; 2) fully-supervised audio-visual segmentation with multiple sound sources, and 3) fully-supervised audio-visual semantic segmentation. The first two settings need to generate binary masks of sounding objects indicating pixels corresponding to the audio, while the third setting further requires to generate semantic maps indicating the object category. To deal with these problems, we propose a new baseline method that uses a temporal pixel-wise audio-visual interaction module to inject audio semantics as guidance for the visual segmentation process. We also design a regularization loss to encourage audio-visual mapping during training. Quantitative and qualitative experiments on the AVSBench dataset compare our approach to several existing methods for related tasks, demonstrating that the proposed method is promising for building a bridge between the audio and pixel-wise visual semantics. Code can be found at https://github.com/OpenNLPLab/AVSBench.

Förderung

  • Major Project of Anhui Province
  • National Key R&D Program of China
  • National Natural Science Foundation of China

Zitieren

APA

Zhou, J., Shen, X., Wang, J., Zhang, J., Sun, W., Zhang, J., Birchfield, S., Guo, D., Kong, L., Wang, M., & Zhong, Y. (2025). Audio-Visual Segmentation with Semantics. International Journal of Computer Vision, 133(4), 1644-1664. https://doi.org/10.1007/s11263-024-02261-x

Vancouver

Zhou J, Shen X, Wang J, Zhang J, Sun W, Zhang J, et al. Audio-Visual Segmentation with Semantics. International Journal of Computer Vision. 2025;133(4):1644-1664. doi:10.1007/s11263-024-02261-x

BibTeX
@article{Zhou2025,
  author = {Zhou, Jinxing and Shen, Xuyang and Wang, Jianyuan and Zhang, Jiayi and Sun, Weixuan and Zhang, Jing and Birchfield, Stan and Guo, Dan and Kong, Lingpeng and Wang, Meng and Zhong, Yiran},
  title = {Audio-Visual Segmentation with Semantics},
  journal = {International Journal of Computer Vision},
  year = {2025},
  volume = {133},
  number = {4},
  pages = {1644--1664},
  doi = {10.1007/s11263-024-02261-x}
}
Kostenlos starten →