EUVIMEDEUVIMED entdecken
Crossref

Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering

Yang Liu, Guanbin Li, Liang Lin

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2023-10 · 45 (10) · 11624-11641

DOI: 10.1109/tpami.2023.3284038 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023-10. Verfasst von Yang Liu und 2 weiteren Autor:innen. Die Arbeit wurde bisher 166-mal zitiert (Crossref) und nennt 94 Literaturangaben. Gefördert von: National Key R&D Program of China, National Natural Science Foundation of China, Basic and Applied Basic Research Foundation of Guangdong Province.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Existing visual question answering methods often suffer from cross-modal spurious correlations and oversimplified event-level reasoning processes that fail to capture event temporality, causality, and dynamics spanning over the video. In this work, to address the task of event-level visual question answering, we propose a framework for cross-modal causal relational reasoning. In particular, a set of causal intervention operations is introduced to discover the underlying causal structures across visual and linguistic modalities. Our framework, named Cross-Modal Causal RelatIonal Reasoning (CMCIR), involves three modules: i) Causality-aware Visual-Linguistic Reasoning (CVLR) module for collaboratively disentangling the visual and linguistic spurious correlations via front-door and back-door causal interventions; ii) Spatial-Temporal Transformer (STT) module for capturing the fine-grained interactions between visual and linguistic semantics; iii) Visual-Linguistic Feature Fusion (VLFF) module for learning the global semantic-aware visual-linguistic representations adaptively. Extensive experiments on four event-level datasets demonstrate the superiority of our CMCIR in discovering visual-linguistic causal structures and achieving robust event-level visual question answering.

Förderung

  • National Key R&D Program of China
  • National Natural Science Foundation of China
  • Basic and Applied Basic Research Foundation of Guangdong Province
  • Guangzhou Science and Technology Planning

Zitieren

APA

Liu, Y., Li, G., & Lin, L. (2023). Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10), 11624-11641. https://doi.org/10.1109/tpami.2023.3284038

Vancouver

Liu Y, Li G, Lin L. Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2023;45(10):11624-11641. doi:10.1109/tpami.2023.3284038

BibTeX
@article{Liu2023,
  author = {Liu, Yang and Li, Guanbin and Lin, Liang},
  title = {Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2023},
  volume = {45},
  number = {10},
  pages = {11624--11641},
  doi = {10.1109/tpami.2023.3284038}
}
Kostenlos starten →