Graph Pooling Inference Network for Text-based VQA
ACM Transactions on Multimedia Computing, Communications, and Applications · 30.04.2024 · 20 (4) · 1-21
DOI: 10.1145/3634918 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 11.01.2024. Verfasst von Sheng Zhou und 4 weiteren Autor:innen. Die Arbeit wurde bisher 19-mal zitiert (Crossref) und nennt 64 Literaturangaben. Gefördert von: National Key Research and Development Program of China, National Natural Science Foundation of China, Major Project of Anhui Province.
Abstract (Originalsprache)
Effectively leveraging objects and optical character recognition (OCR) tokens to reason out pivotal scene text is critical for the challenging Text-based Visual Question Answering (TextVQA) task. Graph-based models can effectively capture the semantic relationship among visual entities (objects and tokens) and report remarkable performance in TextVQA. However, previous efforts usually leverage all visual entities and ignore the negative effect of superfluous entities. This article presents a Graph Pooling Inference Network (GPIN), which is an evolutionary graph learning method to purify the visual entities and capture the core semantics. It is observed that the dense distribution of reduplicative objects and the crowd of semantically dependent OCR tokens usually co-exist in the image. Motivated by this, GPIN adopts an adaptive node dropping strategy to dynamically downscale semantically closed nodes for graph evolution and update. To deepen the comprehension of scene text, GPIN is a dual-path hierarchical graph architecture that progressively aggregates the evolved object graph and the evolved token graph semantics into a graph vector that serves as visual cues to facilitate the answer reasoning. It can effectively eliminate object redundancy and enhance the association of semantically continuous tokens. Experiments conducted on TextVQA and ST-VQA datasets show that GPIN achieves promising performance compared with state-of-the-art methods.
Förderung
- National Key Research and Development Program of China
- National Natural Science Foundation of China
- Major Project of Anhui Province
- University Synergy Innovation Program of Anhui Province
Zitieren
APA
Zhou, S., Guo, D., Yang, X., Dong, J., & Wang, M. (2024). Graph Pooling Inference Network for Text-based VQA. ACM Transactions on Multimedia Computing, Communications, and Applications, 20(4), 1-21. https://doi.org/10.1145/3634918
Vancouver
Zhou S, Guo D, Yang X, Dong J, Wang M. Graph Pooling Inference Network for Text-based VQA. ACM Transactions on Multimedia Computing, Communications, and Applications. 2024;20(4):1-21. doi:10.1145/3634918
BibTeX
@article{Zhou2024,
author = {Zhou, Sheng and Guo, Dan and Yang, Xun and Dong, Jianfeng and Wang, Meng},
title = {Graph Pooling Inference Network for Text-based VQA},
journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
year = {2024},
volume = {20},
number = {4},
pages = {1--21},
doi = {10.1145/3634918}
}
EUVIMED