Nontrivial Audio--Visual Segmentation
IEEE Transactions on Circuits and Systems for Video Technology · 2026-11 · 36 (11) · 17789-17802
DOI: 10.1109/tcsvt.2026.3722652 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems for Video Technology, 2026-11. Verfasst von Xuyi Fan und 5 weiteren Autor:innen. Die Arbeit nennt 60 Literaturangaben. Gefördert von: National Key Research and Development Program of China, National Natural Science Foundation of China, Aeronautical Science Foundation of China.
Abstract (Originalsprache)
Audio–visual matching has become increasingly central in multimodal perception, and audio-visual segmentation (AVS) represents one of its most active research directions. However, existing AVS approaches typically construct acoustic features by weighted averaging and temporal repetition in the time–frequency domain to match the visual feature dimensions for feature mixture, which substantially trivializes the audio representation and limits the effectiveness of cross-modal interaction. To address this limitation, this work replaces the conventional feature-mixture strategy with a lightweight yet principled global temporal audio–visual aligner (GTAVA) to avoid the direct fusion of heterogeneous temporal features. Thus, GTAVA improves computational efficiency while aligning the global temporal behaviors of audio and visual modalities via a normalized-Laplacian multi-scale heat-kernel distance. This alignment enables the two modalities to cooperate more coherently along the video sequence, thereby yielding more discriminative audio queries and enhancing target object segmentation. To further enrich the acoustic features, a hierarchical audio backbone, in which multiple time-frequency resolutions are utilized to generate hierarchical Mel-spectrograms as audio inputs, is proposed in this work. Extensive experiments conducted on three AVS benchmarks demonstrate the state-of-the-art performance of the proposed methods. The source code is released in https://github.com/OrigamiSL/NTAVS
Förderung
- National Key Research and Development Program of China
- National Natural Science Foundation of China
- Aeronautical Science Foundation of China
Zitieren
APA
Fan, X., Li, H., Wang, Y., Qiu, H., Dai, W., & Shen, L. (2026). Nontrivial Audio--Visual Segmentation. IEEE Transactions on Circuits and Systems for Video Technology, 36(11), 17789-17802. https://doi.org/10.1109/tcsvt.2026.3722652
Vancouver
Fan X, Li H, Wang Y, Qiu H, Dai W, Shen L. Nontrivial Audio--Visual Segmentation. IEEE Transactions on Circuits and Systems for Video Technology. 2026;36(11):17789-17802. doi:10.1109/tcsvt.2026.3722652
BibTeX
@article{Fan2026,
author = {Fan, Xuyi and Li, Hongguang and Wang, Yangzhu and Qiu, Huaxin and Dai, Wenhao and Shen, Li},
title = {Nontrivial Audio--Visual Segmentation},
journal = {IEEE Transactions on Circuits and Systems for Video Technology},
year = {2026},
volume = {36},
number = {11},
pages = {17789--17802},
doi = {10.1109/tcsvt.2026.3722652}
}
EUVIMED