EUVIMEDEUVIMED entdecken
Crossref

Learning Alignments of Human Gaze and Fine-grained Task Descriptions

Takumi Nishiyasu, Zhiming Hu, Andreas Bulling, Yoichi Sato

Proceedings of the ACM on Computer Graphics and Interactive Techniques · 01.06.2026 · 9 (2) · 1-18

DOI: 10.1145/3803535 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the ACM on Computer Graphics and Interactive Techniques, online erschienen am 01.05.2026. Verfasst von Takumi Nishiyasu und 3 weiteren Autor:innen. Die Arbeit nennt 68 Literaturangaben. Gefördert von: JST Adopting Sustainable Partnerships for Innovative Research Ecosystem, SPRING.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We propose GTANet – a novel approach to learning the alignments between human gaze scanpaths and fine-grained task descriptions in vision-language tasks. While the influence of tasks on gaze is well known, the relationship between gaze scanpaths and fine-grained task descriptions remains largely unexplored. GTANet addresses this gap by aligning encoded spatiotemporal gaze features with text descriptions. We utilize a patch-based gaze encoder to generate gaze features that reflect visual contexts, and a multimodal feature mixer to fuse the gaze features and the task descriptions, capturing cross-modal alignment. To validate our method, we introduce two novel tasks: gaze-to-question and question-to-gaze retrieval. Experiments on the AiR and MHUG datasets demonstrate that GTANet consistently outperforms baseline methods across all Recall@K metrics, achieving substantial improvements in both retrieval directions. These results confirm the strong link between human gaze and fine-grained task descriptions, thus validating the effectiveness of our approach.

Förderung

  • JST Adopting Sustainable Partnerships for Innovative Research Ecosystem
  • SPRING

Zitieren

APA

Nishiyasu, T., Hu, Z., Bulling, A., & Sato, Y. (2026). Learning Alignments of Human Gaze and Fine-grained Task Descriptions. Proceedings of the ACM on Computer Graphics and Interactive Techniques, 9(2), 1-18. https://doi.org/10.1145/3803535

Vancouver

Nishiyasu T, Hu Z, Bulling A, Sato Y. Learning Alignments of Human Gaze and Fine-grained Task Descriptions. Proceedings of the ACM on Computer Graphics and Interactive Techniques. 2026;9(2):1-18. doi:10.1145/3803535

BibTeX
@article{Nishiyasu2026,
  author = {Nishiyasu, Takumi and Hu, Zhiming and Bulling, Andreas and Sato, Yoichi},
  title = {Learning Alignments of Human Gaze and Fine-grained Task Descriptions},
  journal = {Proceedings of the ACM on Computer Graphics and Interactive Techniques},
  year = {2026},
  volume = {9},
  number = {2},
  pages = {1--18},
  doi = {10.1145/3803535}
}
Kostenlos starten →