Proposal With Alignment: A Bi-Directional Transformer for 360° Video Viewport Proposal
IEEE Transactions on Circuits and Systems for Video Technology · 2024-11 · 34 (11) · 11423-11437
DOI: 10.1109/tcsvt.2024.3419910 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems for Video Technology, 2024-11. Verfasst von Yichen Guo und 6 weiteren Autor:innen. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 57 Literaturangaben. Gefördert von: National Natural Science Foundation of China, China Association for Science and Technology (CAST) through the Young Elite Scientists Sponsorship Program, Alibaba Innovative Research.
Abstract (Originalsprache)
People normally watch 360 ° videos through a head-mounted display, inside which only the content of viewports can be seen. Therefore, viewport proposal, referring to detecting potential viewport candidates, plays an important role in many 360 ° video processing tasks. In this paper, we advance the viewport proposal by further aligning the predicted viewports across frames for individual subject. This provides a better methodology and a deeper perspective to learn the human perceptual behaviours on 360 ° videos. Specifically, we first analyze three 360 ° video datasets and obtain several findings on human consistency, objectness and motion of viewports. Inspired by these findings, we propose a bi-directional transformer approach, named BiT, for 360 ° video viewport proposal and alignment. Specifically, BiT is composed of a multi-level residual module, a bi-directional encoder-decoder module and a spherical matching module. This way, the viewports can be well proposed and aligned via considering multi-level, bi-directional and non-local information. Moreover, the aligned viewports by BiT are used to refine the viewports and improve viewport proposal accuracy in return. Finally, we validate that our BiT approach is superior on viewport proposal, compared with the state-of-the-art approaches. Besides, the aligned viewports from BiT is verified to be effective in multiple applications, such as saliency prediction, trajectory prediction and perceptual video compression.
Förderung
- National Natural Science Foundation of China
- China Association for Science and Technology (CAST) through the Young Elite Scientists Sponsorship Program
- Alibaba Innovative Research
- Beijing Natural Science Foundation
Zitieren
APA
Guo, Y., Xu, M., Jiang, L., Deng, X., Zhou, J., Chen, G., & Sigal, L. (2024). Proposal With Alignment: A Bi-Directional Transformer for 360° Video Viewport Proposal. IEEE Transactions on Circuits and Systems for Video Technology, 34(11), 11423-11437. https://doi.org/10.1109/tcsvt.2024.3419910
Vancouver
Guo Y, Xu M, Jiang L, Deng X, Zhou J, Chen G, et al. Proposal With Alignment: A Bi-Directional Transformer for 360° Video Viewport Proposal. IEEE Transactions on Circuits and Systems for Video Technology. 2024;34(11):11423-11437. doi:10.1109/tcsvt.2024.3419910
BibTeX
@article{Guo2024,
author = {Guo, Yichen and Xu, Mai and Jiang, Lai and Deng, Xin and Zhou, Jing and Chen, Gaoxing and Sigal, Leonid},
title = {Proposal With Alignment: A Bi-Directional Transformer for 360° Video Viewport Proposal},
journal = {IEEE Transactions on Circuits and Systems for Video Technology},
year = {2024},
volume = {34},
number = {11},
pages = {11423--11437},
doi = {10.1109/tcsvt.2024.3419910}
}
EUVIMED