CLIP-Powered TASS: Target-Aware Single-Stream Network for Audio-Visual Question Answering
International Journal of Computer Vision · 2025-05 · 133 (5) · 2581-2598
DOI: 10.1007/s11263-024-02289-z ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Computer Vision, online erschienen am 05.12.2024. Verfasst von Yuanyuan Jiang und 1 weiteren Autor:innen. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 47 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Fundamental Research Funds for the Central Universities, Natural Science Foundation of Hainan Province.
Abstract (Originalsprache)
While vision-language pretrained models (VLMs) excel in various multimodal understanding tasks, their potential in fine-grained audio-visual reasoning, particularly for audio-visual question answering (AVQA), remains largely unexplored. AVQA presents specific challenges for VLMs due to the requirement of visual understanding at the region level and seamless integration with audio modality. Previous VLM-based AVQA methods merely used CLIP as a feature encoder but underutilized its knowledge, and mistreated audio and video as separate entities in a dual-stream framework as most AVQA methods. This paper proposes a new CLIP-powered target-aware single-stream (TASS) network for AVQA using the pretrained knowledge of the CLIP model through the audio-visual matching characteristic of nature. It consists of two key components: the target-aware spatial grounding module (TSG+) and the single-stream joint temporal grounding module (JTG). Specifically, TSG+ module transfers the image-text matching knowledge from CLIP models to the required region-text matching process without corresponding ground-truth labels. Moreover, unlike previous separate dual-stream networks that still required an additional audio-visual fusion module, JTG unifies audio-visual fusion and question-aware temporal grounding in a simplified single-stream architecture. It treats audio and video as a cohesive entity and further extends the image-text matching knowledge to audio-text matching by preserving their temporal correlation with our proposed cross-modal synchrony (CMS) loss. Besides, we propose a simple yet effective preprocessing strategy to optimize accuracy-efficiency trade-offs. Extensive experiments conducted on the MUSIC-AVQA benchmark verified the effectiveness of our proposed method over existing state-of-the-art methods. The code is available at https://github.com/Bravo5542/CLIP-TASS.
Förderung
- National Natural Science Foundation of China
- Fundamental Research Funds for the Central Universities
- Natural Science Foundation of Hainan Province
Zitieren
APA
Jiang, Y., & Yin, J. (2025). CLIP-Powered TASS: Target-Aware Single-Stream Network for Audio-Visual Question Answering. International Journal of Computer Vision, 133(5), 2581-2598. https://doi.org/10.1007/s11263-024-02289-z
Vancouver
Jiang Y, Yin J. CLIP-Powered TASS: Target-Aware Single-Stream Network for Audio-Visual Question Answering. International Journal of Computer Vision. 2025;133(5):2581-2598. doi:10.1007/s11263-024-02289-z
BibTeX
@article{Jiang2025,
author = {Jiang, Yuanyuan and Yin, Jianqin},
title = {CLIP-Powered TASS: Target-Aware Single-Stream Network for Audio-Visual Question Answering},
journal = {International Journal of Computer Vision},
year = {2025},
volume = {133},
number = {5},
pages = {2581--2598},
doi = {10.1007/s11263-024-02289-z}
}
EUVIMED