STAU: A SpatioTemporal-Aware Unit for Video Prediction and Beyond
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-09 · 47 (9) · 7916-7929
DOI: 10.1109/tpami.2025.3572735 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-09. Verfasst von Zheng Chang und 4 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 61 Literaturangaben. Gefördert von: Funds for International Cooperation and Exchange of the National Natural Science Foundation of China, Key Program of the National Natural Science Foundation of China, National Natural Science Foundation of China.
Abstract (Originalsprache)
Video prediction aims to predict future frames by modeling the complex spatiotemporal dynamics in videos. However, most existing methods only model the temporal information and the spatial information for videos in an independent manner but have not fully explored the correlations between both terms. In this paper, we propose a SpatioTemporal-Aware Unit (STAU) for video prediction and beyond by exploring the significant spatiotemporal correlations in videos. On the one hand, the motion-aware attention weights are learned from the spatial states to help aggregate the temporal states in the temporal domain. On the other hand, the appearance-aware attention weights are learned from the temporal states to help aggregate the spatial states in the spatial domain. In this way, the temporal information and the spatial information can be greatly aware of each other in both domains, during which, the spatiotemporal receptive field can also be greatly broadened for more reliable spatiotemporal modeling. Experiments are not only conducted on video prediction tasks (deterministic and stochastic), but also another task beyond video prediction, the early action recognition task. Experimental results show that the proposed STAU can achieve satisfactory performance on all tasks compared with other methods.
Förderung
- Funds for International Cooperation and Exchange of the National Natural Science Foundation of China
- Key Program of the National Natural Science Foundation of China
- National Natural Science Foundation of China
- PCL-CMCC Foundation for science and innovation
Zitieren
APA
Chang, Z., Zhang, X., Wang, S., Ma, S., & Gao, W. (2025). STAU: A SpatioTemporal-Aware Unit for Video Prediction and Beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(9), 7916-7929. https://doi.org/10.1109/tpami.2025.3572735
Vancouver
Chang Z, Zhang X, Wang S, Ma S, Gao W. STAU: A SpatioTemporal-Aware Unit for Video Prediction and Beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(9):7916-7929. doi:10.1109/tpami.2025.3572735
BibTeX
@article{Chang2025,
author = {Chang, Zheng and Zhang, Xinfeng and Wang, Shanshe and Ma, Siwei and Gao, Wen},
title = {STAU: A SpatioTemporal-Aware Unit for Video Prediction and Beyond},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2025},
volume = {47},
number = {9},
pages = {7916--7929},
doi = {10.1109/tpami.2025.3572735}
}
EUVIMED