Towards More Faithful Natural Language Explanation Using Multi-Level Contrastive Learning in VQA
Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (3) · 2849-2857
DOI: 10.1609/aaai.v38i3.28065 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von Chengen Lai und 5 weiteren Autor:innen. Die Arbeit wurde bisher 10-mal zitiert (Crossref).
Abstract (Originalsprache)
Natural language explanation in visual question answer (VQA-NLE) aims to explain the decision-making process of models by generating natural language sentences to increase users' trust in the black-box systems. Existing post-hoc methods have achieved significant progress in obtaining a plausible explanation. However, such post-hoc explanations are not always aligned with human logical inference, suffering from the issues on: 1) Deductive unsatisfiability, the generated explanations do not logically lead to the answer; 2) Factual inconsistency, the model falsifies its counterfactual explanation for answers without considering the facts in images; and 3) Semantic perturbation insensitivity, the model can not recognize the semantic changes caused by small perturbations. These problems reduce the faithfulness of explanations generated by models. To address the above issues, we propose a novel self-supervised Multi-level Contrastive Learning based natural language Explanation model (MCLE) for VQA with semantic-level, image-level, and instance-level factual and counterfactual samples. MCLE extracts discriminative features and aligns the feature spaces from explanations with visual question and answer to generate more consistent explanations. We conduct extensive experiments, ablation analysis, and case study to demonstrate the effectiveness of our method on two VQA-NLE benchmarks.
Zitieren
APA
Lai, C., Song, S., Meng, S., Li, J., Yan, S., & Hu, G. (2024). Towards More Faithful Natural Language Explanation Using Multi-Level Contrastive Learning in VQA. Proceedings of the AAAI Conference on Artificial Intelligence, 38(3), 2849-2857. https://doi.org/10.1609/aaai.v38i3.28065
Vancouver
Lai C, Song S, Meng S, Li J, Yan S, Hu G. Towards More Faithful Natural Language Explanation Using Multi-Level Contrastive Learning in VQA. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(3):2849-2857. doi:10.1609/aaai.v38i3.28065
BibTeX
@article{Lai2024,
author = {Lai, Chengen and Song, Shengli and Meng, Shiqi and Li, Jingyang and Yan, Sitong and Hu, Guangneng},
title = {Towards More Faithful Natural Language Explanation Using Multi-Level Contrastive Learning in VQA},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2024},
volume = {38},
number = {3},
pages = {2849--2857},
doi = {10.1609/aaai.v38i3.28065}
}
EUVIMED