Voxel-Based Multi-Scale Transformer Network for Event Stream Processing
IEEE Transactions on Circuits and Systems for Video Technology · 2024-04 · 34 (4) · 2112-2124
DOI: 10.1109/tcsvt.2023.3301176 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems for Video Technology, 2024-04. Verfasst von Daikun Liu und 2 weiteren Autor:innen. Die Arbeit wurde bisher 18-mal zitiert (Crossref) und nennt 54 Literaturangaben. Gefördert von: National Natural Science Foundation of China, ZhiShan Scholar Program of Southeast University.
Abstract (Originalsprache)
Event cameras are bio-inspired dynamic vision sensors that are superior to frame-based cameras in terms of low power consumption, high dynamic range, and high temporal resolution in computer vision tasks. Recent advances in voxel-based representation learning have successfully exploited the sparsity of events with low computational complexity, but face challenges in extracting spatio-temporal features within voxels and representative global dependencies between voxels, thus limiting their representation power. In this work, towards a better trade-off between accuracy and computation overhead, we propose a novel voxel-based multi-scale transformer network (VMST-Net) to process event streams. Specifically, VMST-Net projects events within voxels into multi-channel frames along the time axis, such that 2D convolutions could be leveraged to encode spatio-temporal features in voxels. Then, VMST-Net utilizes a novel multi-scale multi-head self-attention (MSMHSA) mechanism with a multi-scale fusion (MSF) module that allows different heads within each layer to attend different scale 3D neighborhoods to adaptively aggregate the coarse-to-fine voxel features with little computational costs and parameters. Moreover, to model effective global features while saving computations, we aggregate features in a local-to-global manner by enlarging the coverage of 3D neighborhoods as the network gets deeper. Extensive experimental results on benchmark datasets demonstrate that our model advances state-of-the-art accuracy with low model complexity and computational complexity in all three visual tasks, including object classification, action recognition, and human pose estimation.
Förderung
- National Natural Science Foundation of China
- ZhiShan Scholar Program of Southeast University
Zitieren
APA
Liu, D., Wang, T., & Sun, C. (2024). Voxel-Based Multi-Scale Transformer Network for Event Stream Processing. IEEE Transactions on Circuits and Systems for Video Technology, 34(4), 2112-2124. https://doi.org/10.1109/tcsvt.2023.3301176
Vancouver
Liu D, Wang T, Sun C. Voxel-Based Multi-Scale Transformer Network for Event Stream Processing. IEEE Transactions on Circuits and Systems for Video Technology. 2024;34(4):2112-2124. doi:10.1109/tcsvt.2023.3301176
BibTeX
@article{Liu2024,
author = {Liu, Daikun and Wang, Teng and Sun, Changyin},
title = {Voxel-Based Multi-Scale Transformer Network for Event Stream Processing},
journal = {IEEE Transactions on Circuits and Systems for Video Technology},
year = {2024},
volume = {34},
number = {4},
pages = {2112--2124},
doi = {10.1109/tcsvt.2023.3301176}
}
EUVIMED