EUVIMEDEUVIMED entdecken
Crossref

Causal-Gaze Zero-Shot Learning (CG-ZSL): Causality-Based Human Attention Integration for Generalizable Visual Recognition

Edwin R. Hancock, Yan Hua Dong

Bulletin of Computer and Data Sciences · 2024

DOI: 10.71448/bcds2451-1 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Bulletin of Computer and Data Sciences, 2024. Verfasst von Edwin R. Hancock und 1 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Human gaze has been widely explored as a supervisory signal for fine-grained recognition and zero-shot learning (ZSL). However, existing models merely align machine attention with human gaze using auxiliary losses, leaving the fundamental causal role of gaze in human decision-making unexplored. This leads to models that correlate with gaze but do not reason through it. We propose Causal-Gaze Zero-Shot Learning (CG-ZSL), the first causal framework that integrates human gaze as an explicit mediator in the visual-semantic reasoning pathway. We formalize the ZSL pipeline as a Structural Causal Model (SCM), where human gaze acts as an intermediate variable linking visual attributes to class-level semantic embeddings. Using this formulation, we derive counterfactual attention invariance, enabling the disentanglement of causal attribute regions from dataset-specific biases. We further introduce a Causal Attention Intervention (CAI) module and a Gaze-Mediated Semantic Alignment (GMSA) mechanism that enforce bidirectional causal consistency between gaze, attributes, and predictions. Experiments on CUB, SUN, and AWA2 datasets show significant improvements over state-of-the-art ZSL and GZSL models, especially under distribution shift and domain generalization settings. Unlike prior attention-alignment systems, CG-ZSL produces human-interpretable, causally-grounded explanations and maintains performance under counterfactual perturbations.

Zitieren

APA

Hancock, E. R., & Dong, Y. H. (2024). Causal-Gaze Zero-Shot Learning (CG-ZSL): Causality-Based Human Attention Integration for Generalizable Visual Recognition. Bulletin of Computer and Data Sciences. https://doi.org/10.71448/bcds2451-1

Vancouver

Hancock ER, Dong YH. Causal-Gaze Zero-Shot Learning (CG-ZSL): Causality-Based Human Attention Integration for Generalizable Visual Recognition. Bulletin of Computer and Data Sciences. 2024. doi:10.71448/bcds2451-1

BibTeX
@article{Hancock2024,
  author = {Hancock, Edwin R. and Dong, Yan Hua},
  title = {Causal-Gaze Zero-Shot Learning (CG-ZSL): Causality-Based Human Attention Integration for Generalizable Visual Recognition},
  journal = {Bulletin of Computer and Data Sciences},
  year = {2024},
  doi = {10.71448/bcds2451-1}
}
Kostenlos starten →