EUVIMEDEUVIMED entdecken
Crossref

Semi-Supervised VQA Multi-Modal Explanation via Self-Critical Learning

Wei Suo, Ji Ma, Mengyang Sun, Hanwang Zhang, Peng Wang, Yanning Zhang, Qi Wu

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026-07 · 48 (7) · 8361-8377

DOI: 10.1109/tpami.2026.3669188 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026-07. Verfasst von Wei Suo und 6 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 87 Literaturangaben. Gefördert von: Natural Science Basic Research Program of Shaanxi Province, National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

VQA explanation task aims to explain the decision-making process of VQA models in a way that is easily understandable to humans. Existing methods mostly use visual location or natural language explanation approaches to generate corresponding rationales. Although significant progress has been made, these frameworks are bottlenecked by the following challenges: 1) Uni-modal paradigm inevitably leads to semantic ambiguity of explanations. 2) The reasoning process cannot be faithfully responded to and suffers from logical inconsistency. 3) Human-annotated explanations are expensive and time-consuming to collect. In this paper, we introduce a new Semi-supervised VQA Multi-modal Explanation (SME) method via self-critical learning, which addresses the above challenges by leveraging both visual and textual explanations to comprehensively reveal the inference process of the model. Meanwhile, in order to improve the logical consistency between answers and rationales, we design a novel self-critical strategy to evaluate candidate explanations based on answer reward scores. More importantly, our method can benefit from a tremendous amount of samples without human-annotated explanations with semi-supervised learning. Extensive automatic measures and human evaluations all show the effectiveness of our method. Finally, the framework achieves a new state-of-the-art performance on the three VQA explanation datasets.

Förderung

  • Natural Science Basic Research Program of Shaanxi Province
  • National Natural Science Foundation of China

Zitieren

APA

Suo, W., Ma, J., Sun, M., Zhang, H., Wang, P., Zhang, Y., & Wu, Q. (2026). Semi-Supervised VQA Multi-Modal Explanation via Self-Critical Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(7), 8361-8377. https://doi.org/10.1109/tpami.2026.3669188

Vancouver

Suo W, Ma J, Sun M, Zhang H, Wang P, Zhang Y, et al. Semi-Supervised VQA Multi-Modal Explanation via Self-Critical Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2026;48(7):8361-8377. doi:10.1109/tpami.2026.3669188

BibTeX
@article{Suo2026,
  author = {Suo, Wei and Ma, Ji and Sun, Mengyang and Zhang, Hanwang and Wang, Peng and Zhang, Yanning and Wu, Qi},
  title = {Semi-Supervised VQA Multi-Modal Explanation via Self-Critical Learning},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2026},
  volume = {48},
  number = {7},
  pages = {8361--8377},
  doi = {10.1109/tpami.2026.3669188}
}
Kostenlos starten →