EUVIMEDEUVIMED entdecken
Crossref

Nontrivial Audio--Visual Segmentation

Xuyi Fan, Hongguang Li, Yangzhu Wang, Huaxin Qiu, Wenhao Dai, Li Shen

IEEE Transactions on Circuits and Systems for Video Technology · 2026-11 · 36 (11) · 17789-17802

DOI: 10.1109/tcsvt.2026.3722652 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems for Video Technology, 2026-11. Verfasst von Xuyi Fan und 5 weiteren Autor:innen. Die Arbeit nennt 60 Literaturangaben. Gefördert von: National Key Research and Development Program of China, National Natural Science Foundation of China, Aeronautical Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Audio–visual matching has become increasingly central in multimodal perception, and audio-visual segmentation (AVS) represents one of its most active research directions. However, existing AVS approaches typically construct acoustic features by weighted averaging and temporal repetition in the time–frequency domain to match the visual feature dimensions for feature mixture, which substantially trivializes the audio representation and limits the effectiveness of cross-modal interaction. To address this limitation, this work replaces the conventional feature-mixture strategy with a lightweight yet principled global temporal audio–visual aligner (GTAVA) to avoid the direct fusion of heterogeneous temporal features. Thus, GTAVA improves computational efficiency while aligning the global temporal behaviors of audio and visual modalities via a normalized-Laplacian multi-scale heat-kernel distance. This alignment enables the two modalities to cooperate more coherently along the video sequence, thereby yielding more discriminative audio queries and enhancing target object segmentation. To further enrich the acoustic features, a hierarchical audio backbone, in which multiple time-frequency resolutions are utilized to generate hierarchical Mel-spectrograms as audio inputs, is proposed in this work. Extensive experiments conducted on three AVS benchmarks demonstrate the state-of-the-art performance of the proposed methods. The source code is released in https://github.com/OrigamiSL/NTAVS

Förderung

  • National Key Research and Development Program of China
  • National Natural Science Foundation of China
  • Aeronautical Science Foundation of China

Zitieren

APA

Fan, X., Li, H., Wang, Y., Qiu, H., Dai, W., & Shen, L. (2026). Nontrivial Audio--Visual Segmentation. IEEE Transactions on Circuits and Systems for Video Technology, 36(11), 17789-17802. https://doi.org/10.1109/tcsvt.2026.3722652

Vancouver

Fan X, Li H, Wang Y, Qiu H, Dai W, Shen L. Nontrivial Audio--Visual Segmentation. IEEE Transactions on Circuits and Systems for Video Technology. 2026;36(11):17789-17802. doi:10.1109/tcsvt.2026.3722652

BibTeX
@article{Fan2026,
  author = {Fan, Xuyi and Li, Hongguang and Wang, Yangzhu and Qiu, Huaxin and Dai, Wenhao and Shen, Li},
  title = {Nontrivial Audio--Visual Segmentation},
  journal = {IEEE Transactions on Circuits and Systems for Video Technology},
  year = {2026},
  volume = {36},
  number = {11},
  pages = {17789--17802},
  doi = {10.1109/tcsvt.2026.3722652}
}
Kostenlos starten →