Learning Alignments of Human Gaze and Fine-grained Task Descriptions
Proceedings of the ACM on Computer Graphics and Interactive Techniques · 01.06.2026 · 9 (2) · 1-18
DOI: 10.1145/3803535 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the ACM on Computer Graphics and Interactive Techniques, online erschienen am 01.05.2026. Verfasst von Takumi Nishiyasu und 3 weiteren Autor:innen. Die Arbeit nennt 68 Literaturangaben. Gefördert von: JST Adopting Sustainable Partnerships for Innovative Research Ecosystem, SPRING.
Abstract (Originalsprache)
We propose GTANet – a novel approach to learning the alignments between human gaze scanpaths and fine-grained task descriptions in vision-language tasks. While the influence of tasks on gaze is well known, the relationship between gaze scanpaths and fine-grained task descriptions remains largely unexplored. GTANet addresses this gap by aligning encoded spatiotemporal gaze features with text descriptions. We utilize a patch-based gaze encoder to generate gaze features that reflect visual contexts, and a multimodal feature mixer to fuse the gaze features and the task descriptions, capturing cross-modal alignment. To validate our method, we introduce two novel tasks: gaze-to-question and question-to-gaze retrieval. Experiments on the AiR and MHUG datasets demonstrate that GTANet consistently outperforms baseline methods across all Recall@K metrics, achieving substantial improvements in both retrieval directions. These results confirm the strong link between human gaze and fine-grained task descriptions, thus validating the effectiveness of our approach.
Förderung
- JST Adopting Sustainable Partnerships for Innovative Research Ecosystem
- SPRING
Zitieren
APA
Nishiyasu, T., Hu, Z., Bulling, A., & Sato, Y. (2026). Learning Alignments of Human Gaze and Fine-grained Task Descriptions. Proceedings of the ACM on Computer Graphics and Interactive Techniques, 9(2), 1-18. https://doi.org/10.1145/3803535
Vancouver
Nishiyasu T, Hu Z, Bulling A, Sato Y. Learning Alignments of Human Gaze and Fine-grained Task Descriptions. Proceedings of the ACM on Computer Graphics and Interactive Techniques. 2026;9(2):1-18. doi:10.1145/3803535
BibTeX
@article{Nishiyasu2026,
author = {Nishiyasu, Takumi and Hu, Zhiming and Bulling, Andreas and Sato, Yoichi},
title = {Learning Alignments of Human Gaze and Fine-grained Task Descriptions},
journal = {Proceedings of the ACM on Computer Graphics and Interactive Techniques},
year = {2026},
volume = {9},
number = {2},
pages = {1--18},
doi = {10.1145/3803535}
}
EUVIMED