EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Which AI Sees Like Us? Investigating the Cognitive Plausibility of Language and Vision Models via Eye-Tracking in Human-Robot Interaction

Khashayar Ghamati iD, Maryam Banitalebi Dehkordi, Abolfazl Zaraki iD

Sensors · 2025

DOI: 10.3390/s25154687 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, 2025. Verfasst von Khashayar Ghamati und 2 weiteren Autor:innen. Beteiligte Einrichtung: School of Physics, Engineering and Computer Science (SPECS), University of Hertfordshire, Hatfield AL10 9AB, UK. Hauptthemen laut MEDLINE: Robotics, Eye-Tracking Technology, Cognition, Language. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 29 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

As large language models (LLMs) and vision–language models (VLMs) become increasingly used in robotics area, a crucial question arises: to what extent do these models replicate human-like cognitive processes, particularly within socially interactive contexts? Whilst these models demonstrate impressive multimodal reasoning and perception capabilities, their cognitive plausibility remains underexplored. In this study, we address this gap by using human visual attention as a behavioural proxy for cognition in a naturalistic human-robot interaction (HRI) scenario. Eye-tracking data were previously collected from participants engaging in social human-human interactions, providing frame-level gaze fixations as a human attentional ground truth. We then prompted a state-of-the-art VLM (LLaVA) to generate scene descriptions, which were processed by four LLMs (DeepSeek-R1-Distill-Qwen-7B, Qwen1.5-7B-Chat, LLaMA-3.1-8b-instruct, and Gemma-7b-it) to infer saliency points. Critically, we evaluated each model in both stateless and memory-augmented (short-term memory, STM) modes to assess the influence of temporal context on saliency prediction. Our results presented that whilst stateless LLaVA most closely replicates human gaze patterns, STM confers measurable benefits only for DeepSeek, whose lexical anchoring mirrors human rehearsal mechanisms. Other models exhibited degraded performance with memory due to prompt interference or limited contextual integration. This work introduces a novel, empirically grounded framework for assessing cognitive plausibility in generative models and underscores the role of short-term memory in shaping human-like visual attention in robotic systems.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

RoboticsEye-Tracking TechnologyCognitionLanguageArtificial IntelligenceVision, OcularHumansAttentionMaleFemaleAdultFixation, OcularEye Movements

Stichwörter der Autor:innen

AI agentLLMsVLMsadaptive AI systemshuman attention modelinghuman-computer interactionhuman-robot interactionpersonalised large language modelsshort-term memory

Autor:innen und Institutionen

  • School of Physics, Engineering and Computer Science (SPECS), University of Hertfordshire, Hatfield AL10 9AB, UK.
  • Robotics Research Group, University of Hertfordshire, Hatfield AL10 9AB, UK.

ORCID-Profile: Khashayar Ghamati, Abolfazl Zaraki

Zitieren

APA

Ghamati, K., Dehkordi, M. B., & Zaraki, A. (2025). Which AI Sees Like Us? Investigating the Cognitive Plausibility of Language and Vision Models via Eye-Tracking in Human-Robot Interaction. Sensors. https://doi.org/10.3390/s25154687

Vancouver

Ghamati K, Dehkordi MB, Zaraki A. Which AI Sees Like Us? Investigating the Cognitive Plausibility of Language and Vision Models via Eye-Tracking in Human-Robot Interaction. Sensors. 2025. doi:10.3390/s25154687

BibTeX
@article{Ghamati2025,
  author = {Ghamati, Khashayar and Dehkordi, Maryam Banitalebi and Zaraki, Abolfazl},
  title = {Which AI Sees Like Us? Investigating the Cognitive Plausibility of Language and Vision Models via Eye-Tracking in Human-Robot Interaction},
  journal = {Sensors},
  year = {2025},
  doi = {10.3390/s25154687}
}
Kostenlos starten →