EUVIMEDEUVIMED entdecken
Crossref

VPFormer: Leveraging Transformer with Voxel Integration for Viewport Prediction in Volumetric Video

Jie Li, Zhixia Zhao, Qiyue Li, Zhixin Li, Pengyuan Zhou, Zhi Liu, Hao Zhou, Zhu Li

ACM Transactions on Multimedia Computing, Communications, and Applications · 30.06.2025 · 21 (6) · 1-27

DOI: 10.1145/3730402 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, 2025-07. Verfasst von Jie Li und 7 weiteren Autor:innen. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 55 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Anhui Provincial Natural Science Foundation, 111 Project.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

With the continuous advancement of computer vision, image processing technologies, volumetric video, represented by point cloud videos, holds the potential for extensive applications in areas such as Virtual Reality (VR) and Augmented Reality (AR). Viewport prediction, also referred to as Field of View (FoV) prediction, is a crucial component in emerging VR and AR applications, playing a vital role in the transmission of point cloud videos. Currently, models for viewpoint prediction that integrate feature extraction and FoV information heavily rely on the spatial-temporal features extracted by convolutional neural networks. However, the drawback of 3D convolution lies in its inability to effectively capture long-term spatial-temporal dependencies within videos. Moreover, the temporal contrast layer used for time feature extraction only compares features within each block, leading to matching errors and inaccurate temporal feature extraction, consequently diminishing predictive performance. To address these limitations, we propose a Transformer-based Volumetric Point Cloud Video Viewport Prediction Network (VPFormer) that can efficiently extract spatial-temporal features from point cloud videos. VPFormer constitutes a viewport prediction framework that combines the spatial-temporal features of point cloud videos with user trajectory information. Specifically, we introduce a novel sampling method that effectively preserves spatial-temporal information while reducing computational complexity. Additionally, we incorporate context-aware dynamic positional encoding to capture inter-frame spatial-temporal context information. Subsequently, we introduce a voxel-based temporal contrast layer and partition the point cloud into smaller voxel blocks during feature matching, significantly reducing matching errors and enhancing the analysis and extraction of temporal features. Finally, by combining the spatial-temporal features of point cloud videos with user head trajectory information, we successfully predict future user viewpoints. Experimental results demonstrate that this approach outperforms other solutions in terms of performance.

Förderung

  • National Natural Science Foundation of China
  • Anhui Provincial Natural Science Foundation
  • 111 Project
  • Fundamental Research Funds for the Central Universities
  • JSPS KAKENHI
  • Telecommunications Advancement Foundation and JST ASPIRE

Zitieren

APA

Li, J., Zhao, Z., Li, Q., Li, Z., Zhou, P., Liu, Z., Zhou, H., & Li, Z. (2025). VPFormer: Leveraging Transformer with Voxel Integration for Viewport Prediction in Volumetric Video. ACM Transactions on Multimedia Computing, Communications, and Applications, 21(6), 1-27. https://doi.org/10.1145/3730402

Vancouver

Li J, Zhao Z, Li Q, Li Z, Zhou P, Liu Z, et al. VPFormer: Leveraging Transformer with Voxel Integration for Viewport Prediction in Volumetric Video. ACM Transactions on Multimedia Computing, Communications, and Applications. 2025;21(6):1-27. doi:10.1145/3730402

BibTeX
@article{Li2025,
  author = {Li, Jie and Zhao, Zhixia and Li, Qiyue and Li, Zhixin and Zhou, Pengyuan and Liu, Zhi and Zhou, Hao and Li, Zhu},
  title = {VPFormer: Leveraging Transformer with Voxel Integration for Viewport Prediction in Volumetric Video},
  journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
  year = {2025},
  volume = {21},
  number = {6},
  pages = {1--27},
  doi = {10.1145/3730402}
}
Kostenlos starten →