Towards Energy-efficient Audio-visual Classification via Multimodal Interactive Spiking Neural Network
ACM Transactions on Multimedia Computing, Communications, and Applications · 31.05.2025 · 21 (5) · 1-24
DOI: 10.1145/3721981 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 22.05.2025. Verfasst von Xu Liu und 4 weiteren Autor:innen. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 63 Literaturangaben. Gefördert von: National Key R&D Program of China, National Natural Science Foundation of China, Ordos Science and Technology Major.
Abstract (Originalsprache)
The Audio-visual Classification (AVC) task aims to determine video categories by integrating audio and visual signals. Traditional methods for AVC leverage Artificial Neural Networks (ANNs) that operate on floating-point features, affording large parameter counts and consuming extensive energy. Recent research has shifted towards brain-inspired Spiking Neural Networks (SNNs), which transmit audiovisual information through sparser 0/1 spike features allowing for better energy efficiency. However, a byproduct of such sparsity is the increased difficulty in effectively encoding and utilizing these spike features. Moreover, the spike firing characteristics based on neuron membrane potential cause asynchronous spike activations due to the heterogeneous distributions of different modalities in the AVC task, resulting in cross-modal asynchronization. This issue is often overlooked by prior SNN models, resulting in lower classification accuracy compared to traditional ANN models. To address these challenges, we present a new Multimodal Interaction Spiking Network (MISNet), the first to successfully balance both accuracy and efficiency for the AVC task. As the core of MISNet, we propose a Multimodal Leaky Integrate-and-fire (MLIF) neuron, which coordinates and synchronizes the spike activations of audiovisual signals within a single neuron, distinguishing it from the prior paradigm of SNNs that relies on multiple separate processing neurons. As a result, our MISNet enables to generate audio and visual spiking features with effective cross-modal fusion. Additionally, we propose to add extra loss regularizations before fusing the obtained audio-visual features for final classification, thereby benefiting unimodal spiking learning for multimodal interaction. We evaluate our method on five audio-visual datasets, demonstrating advanced performance in both accuracy and energy consumption.
Förderung
- National Key R&D Program of China
- National Natural Science Foundation of China
- Ordos Science and Technology Major
- State Grid Co., Ltd. Headquarters Technology Project
- Major Project of Anhui Province
- Anhui Provincial Natural Science Foundation—Distinguished Young Scholars Project
- Fundamental Research Funds for the Central Universities
Zitieren
APA
Liu, X., Xia, N., Zhou, J., Li, Z., & Guo, D. (2025). Towards Energy-efficient Audio-visual Classification via Multimodal Interactive Spiking Neural Network. ACM Transactions on Multimedia Computing, Communications, and Applications, 21(5), 1-24. https://doi.org/10.1145/3721981
Vancouver
Liu X, Xia N, Zhou J, Li Z, Guo D. Towards Energy-efficient Audio-visual Classification via Multimodal Interactive Spiking Neural Network. ACM Transactions on Multimedia Computing, Communications, and Applications. 2025;21(5):1-24. doi:10.1145/3721981
BibTeX
@article{Liu2025,
author = {Liu, Xu and Xia, Na and Zhou, Jinxing and Li, Zhangbin and Guo, Dan},
title = {Towards Energy-efficient Audio-visual Classification via Multimodal Interactive Spiking Neural Network},
journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
year = {2025},
volume = {21},
number = {5},
pages = {1--24},
doi = {10.1145/3721981}
}
EUVIMED