EUVIMEDEUVIMED entdecken
Crossref

In the Eye of the Beholder: Gaze and Actions in First Person Video

Yin Li, Miao Liu, James M Rehg

IEEE Transactions on Pattern Analysis and Machine Intelligence · 01.06.2023 · 45 (6) · 6731-6747

DOI: 10.1109/tpami.2021.3051319 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, erschienen am 01.06.2023. Verfasst von Yin Li und 2 weiteren Autor:innen. Hauptthemen laut MEDLINE: Algorithms, Hand. Die Arbeit wurde bisher 57-mal zitiert (Crossref) und nennt 117 Literaturangaben. Gefördert von: NIBIB NIH HHS, National Institute of Biomedical Imaging and Bioengineering, Intel Science of Technology Center for Pervasive Computing.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We address the task of jointly determining what a person is doing and where they are looking based on the analysis of video captured by a headworn camera. To facilitate our research, we first introduce the EGTEA Gaze+ dataset. Our dataset comes with videos, gaze tracking data, hand masks and action annotations, thereby providing the most comprehensive benchmark for First Person Vision (FPV). Moving beyond the dataset, we propose a novel deep model for joint gaze estimation and action recognition in FPV. Our method describes the participant's gaze as a probabilistic variable and models its distribution using stochastic units in a deep network. We further sample from these stochastic units, generating an attention map to guide the aggregation of visual features for action recognition. Our method is evaluated on our EGTEA Gaze+ dataset and achieves a performance level that exceeds the state-of-the-art by a significant margin. More importantly, we demonstrate that our model can be applied to larger scale FPV dataset-EPIC-Kitchens even without using gaze, offering new state-of-the-art results on FPV action recognition.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

AlgorithmsHandHumansAttentionVideo RecordingFixation, Ocular

Förderung

  • NIBIB NIH HHS
  • National Institute of Biomedical Imaging and Bioengineering
  • Intel Science of Technology Center for Pervasive Computing

Zitieren

APA

Li, Y., Liu, M., & Rehg, J. M. (2023). In the Eye of the Beholder: Gaze and Actions in First Person Video. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(6), 6731-6747. https://doi.org/10.1109/tpami.2021.3051319

Vancouver

Li Y, Liu M, Rehg JM. In the Eye of the Beholder: Gaze and Actions in First Person Video. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2023;45(6):6731-6747. doi:10.1109/tpami.2021.3051319

BibTeX
@article{Li2023,
  author = {Li, Yin and Liu, Miao and Rehg, James M},
  title = {In the Eye of the Beholder: Gaze and Actions in First Person Video},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2023},
  volume = {45},
  number = {6},
  pages = {6731--6747},
  doi = {10.1109/tpami.2021.3051319}
}
Kostenlos starten →