Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2023-10 · 45 (10) · 11624-11641
DOI: 10.1109/tpami.2023.3284038 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023-10. Verfasst von Yang Liu und 2 weiteren Autor:innen. Die Arbeit wurde bisher 166-mal zitiert (Crossref) und nennt 94 Literaturangaben. Gefördert von: National Key R&D Program of China, National Natural Science Foundation of China, Basic and Applied Basic Research Foundation of Guangdong Province.
Abstract (Originalsprache)
Existing visual question answering methods often suffer from cross-modal spurious correlations and oversimplified event-level reasoning processes that fail to capture event temporality, causality, and dynamics spanning over the video. In this work, to address the task of event-level visual question answering, we propose a framework for cross-modal causal relational reasoning. In particular, a set of causal intervention operations is introduced to discover the underlying causal structures across visual and linguistic modalities. Our framework, named Cross-Modal Causal RelatIonal Reasoning (CMCIR), involves three modules: i) Causality-aware Visual-Linguistic Reasoning (CVLR) module for collaboratively disentangling the visual and linguistic spurious correlations via front-door and back-door causal interventions; ii) Spatial-Temporal Transformer (STT) module for capturing the fine-grained interactions between visual and linguistic semantics; iii) Visual-Linguistic Feature Fusion (VLFF) module for learning the global semantic-aware visual-linguistic representations adaptively. Extensive experiments on four event-level datasets demonstrate the superiority of our CMCIR in discovering visual-linguistic causal structures and achieving robust event-level visual question answering.
Förderung
- National Key R&D Program of China
- National Natural Science Foundation of China
- Basic and Applied Basic Research Foundation of Guangdong Province
- Guangzhou Science and Technology Planning
Zitieren
APA
Liu, Y., Li, G., & Lin, L. (2023). Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10), 11624-11641. https://doi.org/10.1109/tpami.2023.3284038
Vancouver
Liu Y, Li G, Lin L. Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2023;45(10):11624-11641. doi:10.1109/tpami.2023.3284038
BibTeX
@article{Liu2023,
author = {Liu, Yang and Li, Guanbin and Lin, Liang},
title = {Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2023},
volume = {45},
number = {10},
pages = {11624--11641},
doi = {10.1109/tpami.2023.3284038}
}
EUVIMED