EUVIMEDEUVIMED entdecken
Crossref

UniAV: Unified Audio-Visual Perception for Multi-Task Video Event Localization

Tiantian Geng, Teng Wang, Jinming Duan, Yanfu Zhang, Weili Guan, Feng Zheng, Ling Shao

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-11 · 47 (11) · 10280-10294

DOI: 10.1109/tpami.2025.3593932 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-11. Verfasst von Tiantian Geng und 6 weiteren Autor:innen. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 81 Literaturangaben. Gefördert von: National Key Research and Development Program of China, National Outstanding Youth Science Fund Project of National Natural Science Foundation of China, British Heart Foundation Manchester Research Excellence Award.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Video event localization tasks include temporal action localization (TAL), sound event detection (SED) and audio-visual event localization (AVEL). Existing methods tend to over-specialize on individual tasks, neglecting the equal importance of these different events for a complete understanding of video content. In this work, we aim to develop a unified framework to solve TAL, SED and AVEL tasks together to facilitate holistic video understanding. However, it is challenging since different tasks emphasize distinct event characteristics and there are substantial disparities in existing task-specific datasets (size/domain/duration). It leads to unsatisfactory results when applying a naive multi-task strategy. To tackle the problem, we introduce UniAV, a Unified Audio-Visual perception network to effectively learn and share mutually beneficial knowledge across tasks and modalities. Concretely, we propose a unified audio-visual encoder to derive generic representations from multiple temporal scales for videos from all tasks. Meanwhile, task-specific experts are designed to capture the unique knowledge specific to each task. Besides, instead of using separate prediction heads, we develop a novel unified language-aware classifier by utilizing semantic-aligned task prompts, enabling our model to flexibly localize various instances across tasks with an impressive open-set ability to localize novel categories. Extensive experiments demonstrate that UniAV, with its unified architecture, significantly outperforms both single-task models and the naive multi-task baseline across all three tasks. It achieves superior or on-par performances compared to the state-of-the-art task-specific methods on ActivityNet 1.3, DESED and UnAV-100 benchmarks.

Förderung

  • National Key Research and Development Program of China
  • National Outstanding Youth Science Fund Project of National Natural Science Foundation of China
  • British Heart Foundation Manchester Research Excellence Award

Zitieren

APA

Geng, T., Wang, T., Duan, J., Zhang, Y., Guan, W., Zheng, F., & Shao, L. (2025). UniAV: Unified Audio-Visual Perception for Multi-Task Video Event Localization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(11), 10280-10294. https://doi.org/10.1109/tpami.2025.3593932

Vancouver

Geng T, Wang T, Duan J, Zhang Y, Guan W, Zheng F, et al. UniAV: Unified Audio-Visual Perception for Multi-Task Video Event Localization. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(11):10280-10294. doi:10.1109/tpami.2025.3593932

BibTeX
@article{Geng2025,
  author = {Geng, Tiantian and Wang, Teng and Duan, Jinming and Zhang, Yanfu and Guan, Weili and Zheng, Feng and Shao, Ling},
  title = {UniAV: Unified Audio-Visual Perception for Multi-Task Video Event Localization},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2025},
  volume = {47},
  number = {11},
  pages = {10280--10294},
  doi = {10.1109/tpami.2025.3593932}
}
Kostenlos starten →