EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

In the Eye of Transformer: Global–Local Correlation for Egocentric Gaze Estimation and Beyond

Bolin Lai iD, Miao Liu, Fiona Ryan, James M Rehg

International Journal of Computer Vision · 2024-03 · 132 (3) · 854-871

DOI: 10.1007/s11263-023-01879-7 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Computer Vision, online erschienen am 18.10.2023. Verfasst von Bolin Lai und 3 weiteren Autor:innen. Beteiligte Einrichtung: Georgia Institute of Technology, Atlanta, GA 30308 USA. Die Arbeit wurde bisher 35-mal zitiert (Crossref) und nennt 89 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NIMH NIH HHS, National Institutes of Health.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Predicting human’s gaze from egocentric videos serves as a critical role for human intention understanding in daily activities. In this paper, we present the first transformer-based model to address the challenging problem of egocentric gaze estimation. We observe that the connection between the global scene context and local visual information is vital for localizing the gaze fixation from egocentric video frames. To this end, we design the transformer encoder to embed the global context as one additional visual token and further propose a novel global–local correlation module to explicitly model the correlation of the global token and each local token. We validate our model on two egocentric video datasets – EGTEA Gaze + and Ego4D. Our detailed ablation studies demonstrate the benefits of our method. In addition, our approach exceeds the previous state-of-the-art model by a large margin. We also apply our model to a novel gaze saccade/fixation prediction task and the traditional action recognition problem. The consistent gains suggest the strong generalization capability of our model. We also provide additional visualizations to support our claim that global–local correlation serves a key representation for predicting gaze fixation from egocentric videos. More details can be found in our website ( https://bolinlai.github.io/GLC-EgoGazeEst ).

Schlagwörter

Stichwörter der Autor:innen

Egocentric gaze estimationGlobal–local correlationVision transformer

Autor:innen und Institutionen

  • Georgia Institute of Technology, Atlanta, GA 30308 USA.
  • Meta AI, Menlo Park, CA 94025 USA.

ORCID-Profile: Bolin Lai

Förderung

  • NIMH NIH HHS
  • National Institutes of Health

Zitieren

APA

Lai, B., Liu, M., Ryan, F., & Rehg, J. M. (2024). In the Eye of Transformer: Global–Local Correlation for Egocentric Gaze Estimation and Beyond. International Journal of Computer Vision, 132(3), 854-871. https://doi.org/10.1007/s11263-023-01879-7

Vancouver

Lai B, Liu M, Ryan F, Rehg JM. In the Eye of Transformer: Global–Local Correlation for Egocentric Gaze Estimation and Beyond. International Journal of Computer Vision. 2024;132(3):854-871. doi:10.1007/s11263-023-01879-7

BibTeX
@article{Lai2024,
  author = {Lai, Bolin and Liu, Miao and Ryan, Fiona and Rehg, James M},
  title = {In the Eye of Transformer: Global–Local Correlation for Egocentric Gaze Estimation and Beyond},
  journal = {International Journal of Computer Vision},
  year = {2024},
  volume = {132},
  number = {3},
  pages = {854--871},
  doi = {10.1007/s11263-023-01879-7}
}
Kostenlos starten →