EUVIMEDEUVIMED entdecken
Crossref

Integrating Neural-Symbolic Reasoning With Variational Causal Inference Network for Explanatory Visual Question Answering

Dizhan Xue, Shengsheng Qian, Changsheng Xu

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-12 · 46 (12) · 7893-7908

DOI: 10.1109/tpami.2024.3398012 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-12. Verfasst von Dizhan Xue und 2 weiteren Autor:innen. Die Arbeit wurde bisher 33-mal zitiert (Crossref) und nennt 89 Literaturangaben. Gefördert von: National Key Research and Development Program of China, Beijing Natural Science Foundation, National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recently, a novel multimodal reasoning task named Explanatory Visual Question Answering (EVQA) has been introduced, which combines answering visual questions with multimodal explanation generation to expound upon the underlying reasoning processes. In contrast to conventional Visual Question Answering (VQA) that merely concentrates on providing answers, EVQA aims to improve the explainability and verifiability of reasoning by providing user-friendly explanations. Despite the improved explainability of inferred results, the existing EVQA models still adopt black-box neural networks to infer results, lacking the explainability of the reasoning process. Moreover, existing EVQA models commonly predict answers and explanations in isolation, overlooking the inherent causal correlation between them. To handle these challenges, we propose a Program-guided Variational Causal Inference Network (Pro-VCIN) that integrates neural-symbolic reasoning with variational causal inference and constructs causal correlations between the predicted answers and explanations. First, we utilize pretrained models to extract visual features and convert questions into the corresponding programs. Second, we propose a multimodal program Transformer to translate programs and the related visual features into coherent and rational explanations of the reasoning processes. Finally, we propose a variational causal inference to construct the target structural causal model and predict answers based on the causal correlation to explanations. Comprehensive experiments conducted on EVQA benchmark datasets reveal the superiority of Pro-VCIN in terms of both performance and explainability over state-of-the-art EVQA methods.

Förderung

  • National Key Research and Development Program of China
  • Beijing Natural Science Foundation
  • National Natural Science Foundation of China

Zitieren

APA

Xue, D., Qian, S., & Xu, C. (2024). Integrating Neural-Symbolic Reasoning With Variational Causal Inference Network for Explanatory Visual Question Answering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12), 7893-7908. https://doi.org/10.1109/tpami.2024.3398012

Vancouver

Xue D, Qian S, Xu C. Integrating Neural-Symbolic Reasoning With Variational Causal Inference Network for Explanatory Visual Question Answering. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(12):7893-7908. doi:10.1109/tpami.2024.3398012

BibTeX
@article{Xue2024,
  author = {Xue, Dizhan and Qian, Shengsheng and Xu, Changsheng},
  title = {Integrating Neural-Symbolic Reasoning With Variational Causal Inference Network for Explanatory Visual Question Answering},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2024},
  volume = {46},
  number = {12},
  pages = {7893--7908},
  doi = {10.1109/tpami.2024.3398012}
}
Kostenlos starten →