EUVIMEDEUVIMED entdecken
Crossref

Multi-Timescale Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learning

Wenrui Li, Penghong Wang, Xingtao Wang, Wangmeng Zuo, Xiaopeng Fan, Yonghong Tian

IEEE Transactions on Circuits and Systems for Video Technology · 2025-11 · 35 (11) · 10772-10786

DOI: 10.1109/tcsvt.2025.3574499 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems for Video Technology, 2025-11. Verfasst von Wenrui Li und 5 weiteren Autor:innen. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 61 Literaturangaben. Gefördert von: National Key Research and Development Program of China, National Natural Science Foundation of China, Fundamental Research Funds for the Central Universities.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Audio-visual zero-shot learning (ZSL) has been extensively researched for its capability to classify video data from unseen classes during training. Nevertheless, current methodologies often struggle with background scene biases and inadequate motion detail. This paper proposes a novel dual-stream Multi-Timescale Motion-Decoupled Spiking Transformer (MDST++), which decouples contextual semantic information and sparse dynamic motion information. The recurrent joint learning unit is proposed to extract contextual semantic information and capture joint knowledge across various modalities to understand the environment of actions. By converting RGB images to events, our method captures motion information more accurately and mitigates background scene biases. Moreover, we introduce a discrepancy analysis block to model audio motion information. To enhance the robustness of SNNs in extracting temporal and motion cues, we dynamically adjust the threshold of Leaky Integrate-and-Fire neurons based on global motion and contextual semantic information. Our experiments validate the effectiveness of MDST++, demonstrating their consistent superiority over state-of-the-art methods on mainstream benchmarks. Additionally, incorporating motion and multi-timescale information significantly improves HM and ZSL accuracy by 26.2% and 39.9%.

Förderung

  • National Key Research and Development Program of China
  • National Natural Science Foundation of China
  • Fundamental Research Funds for the Central Universities

Zitieren

APA

Li, W., Wang, P., Wang, X., Zuo, W., Fan, X., & Tian, Y. (2025). Multi-Timescale Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learning. IEEE Transactions on Circuits and Systems for Video Technology, 35(11), 10772-10786. https://doi.org/10.1109/tcsvt.2025.3574499

Vancouver

Li W, Wang P, Wang X, Zuo W, Fan X, Tian Y. Multi-Timescale Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learning. IEEE Transactions on Circuits and Systems for Video Technology. 2025;35(11):10772-10786. doi:10.1109/tcsvt.2025.3574499

BibTeX
@article{Li2025,
  author = {Li, Wenrui and Wang, Penghong and Wang, Xingtao and Zuo, Wangmeng and Fan, Xiaopeng and Tian, Yonghong},
  title = {Multi-Timescale Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learning},
  journal = {IEEE Transactions on Circuits and Systems for Video Technology},
  year = {2025},
  volume = {35},
  number = {11},
  pages = {10772--10786},
  doi = {10.1109/tcsvt.2025.3574499}
}
Kostenlos starten →