Toward Visual Behavior and Attention Understanding for Augmented 360 Degree Videos
ACM Transactions on Multimedia Computing, Communications, and Applications · 30.06.2023 · 19 (2s) · 1-24
DOI: 10.1145/3565024 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 17.02.2023. Verfasst von Yucheng Zhu und 7 weiteren Autor:innen. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 68 Literaturangaben. Gefördert von: National Natural Science Foundation of China, National Key R&D Program of China, China Postdoctoral Science Foundation.
Abstract (Originalsprache)
Augmented reality (AR) overlays digital content onto reality. In an AR system, correct and precise estimations of user visual fixations and head movements can enhance the quality of experience by allocating more computational resources for analyzing, rendering, and 3D registration on the areas of interest. However, there is inadequate research to help in understanding the visual explorations of the users when using an AR system or modeling AR visual attention. To bridge the gap between the saliency prediction on real-world scenes and on scenes augmented by virtual information, we construct the ARVR saliency dataset. The virtual reality (VR) technique is employed to simulate the real-world. Annotations of object recognition and tracking as augmented contents are blended into omnidirectional videos. The saliency annotations of head and eye movements for both original and augmented videos are collected and together constitute the ARVR dataset. We also design a model that is capable of solving the saliency prediction problem in AR. Local block images are extracted to simulate the viewport and offset the projection distortion. Conspicuous visual cues in the local block images are extracted to constitute the spatial features. The optical flow information is estimated as an important temporal feature. We also consider the interplay between virtual information and reality. The composition of the augmentation information is distinguished, and the joint effects of adversarial augmentation and complementary augmentation are estimated. The Markov chain is constructed with block images as graph nodes. In the determination of the edge weights, both the characteristics of the viewing behaviors and the visual saliency mechanisms are considered. The order of importance for block images is estimated through the state of equilibrium of the Markov chain. Extensive experiments are conducted to demonstrate the effectiveness of the proposed method.
Förderung
- National Natural Science Foundation of China
- National Key R&D Program of China
- China Postdoctoral Science Foundation
Zitieren
APA
Zhu, Y., Min, X., Zhu, D., Zhai, G., Yang, X., Zhang, W., Gu, K., & Zhou, J. (2023). Toward Visual Behavior and Attention Understanding for Augmented 360 Degree Videos. ACM Transactions on Multimedia Computing, Communications, and Applications, 19(2s), 1-24. https://doi.org/10.1145/3565024
Vancouver
Zhu Y, Min X, Zhu D, Zhai G, Yang X, Zhang W, et al. Toward Visual Behavior and Attention Understanding for Augmented 360 Degree Videos. ACM Transactions on Multimedia Computing, Communications, and Applications. 2023;19(2s):1-24. doi:10.1145/3565024
BibTeX
@article{Zhu2023,
author = {Zhu, Yucheng and Min, Xiongkuo and Zhu, Dandan and Zhai, Guangtao and Yang, Xiaokang and Zhang, Wenjun and Gu, Ke and Zhou, Jiantao},
title = {Toward Visual Behavior and Attention Understanding for Augmented 360 Degree Videos},
journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
year = {2023},
volume = {19},
number = {2s},
pages = {1--24},
doi = {10.1145/3565024}
}
EUVIMED