Point Spatio-Temporal Transformer Networks for Point Cloud Video Modeling
IEEE Transactions on Pattern Analysis and Machine Intelligence · 01.02.2023 · 45 (2) · 2181-2192
DOI: 10.1109/tpami.2022.3161735 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, erschienen am 01.02.2023. Verfasst von Hehe Fan und 2 weiteren Autor:innen. Die Arbeit wurde bisher 102-mal zitiert (Crossref) und nennt 60 Literaturangaben. Gefördert von: Agency for Science, Technology and Research.
Abstract (Originalsprache)
Due to the inherent unorderliness and irregularity of point cloud, points emerge inconsistently across different frames in a point cloud video. To capture the dynamics in point cloud videos, tracking points and limiting temporal modeling range are usually employed to preserve spatio-temporal structure. However, as points may flow in and out across frames, computing accurate point trajectories is extremely difficult, especially for long videos. Moreover, when points move fast, even in a small temporal window, points may still escape from a region. Besides, using the same temporal range for different motions may not accurately capture the temporal structure. In this paper, we propose a Point Spatio-Temporal Transformer (PST-Transformer). To preserve the spatio-temporal structure, PST-Transformer adaptively searches related or similar points across the entire video by performing self-attention on point features. Moreover, our PST-Transformer is equipped with an ability to encode spatio-temporal structure. Because point coordinates are irregular and unordered but point timestamps exhibit regularities and order, the spatio-temporal encoding is decoupled to reduce the impact of the spatial irregularity on the temporal modeling. By properly preserving and encoding spatio-temporal structure, our PST-Transformer effectively models point cloud videos and shows superior performance on 3D action recognition and 4D semantic segmentation.
Förderung
- Agency for Science, Technology and Research
Zitieren
APA
Fan, H., Yang, Y., & Kankanhalli, M. (2023). Point Spatio-Temporal Transformer Networks for Point Cloud Video Modeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2), 2181-2192. https://doi.org/10.1109/tpami.2022.3161735
Vancouver
Fan H, Yang Y, Kankanhalli M. Point Spatio-Temporal Transformer Networks for Point Cloud Video Modeling. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2023;45(2):2181-2192. doi:10.1109/tpami.2022.3161735
BibTeX
@article{Fan2023,
author = {Fan, Hehe and Yang, Yi and Kankanhalli, Mohan},
title = {Point Spatio-Temporal Transformer Networks for Point Cloud Video Modeling},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2023},
volume = {45},
number = {2},
pages = {2181--2192},
doi = {10.1109/tpami.2022.3161735}
}
EUVIMED