EUVIMEDEUVIMED entdecken
Crossref

Graph Pooling Inference Network for Text-based VQA

Sheng Zhou, Dan Guo, Xun Yang, Jianfeng Dong, Meng Wang

ACM Transactions on Multimedia Computing, Communications, and Applications · 30.04.2024 · 20 (4) · 1-21

DOI: 10.1145/3634918 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 11.01.2024. Verfasst von Sheng Zhou und 4 weiteren Autor:innen. Die Arbeit wurde bisher 19-mal zitiert (Crossref) und nennt 64 Literaturangaben. Gefördert von: National Key Research and Development Program of China, National Natural Science Foundation of China, Major Project of Anhui Province.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Effectively leveraging objects and optical character recognition (OCR) tokens to reason out pivotal scene text is critical for the challenging Text-based Visual Question Answering (TextVQA) task. Graph-based models can effectively capture the semantic relationship among visual entities (objects and tokens) and report remarkable performance in TextVQA. However, previous efforts usually leverage all visual entities and ignore the negative effect of superfluous entities. This article presents a Graph Pooling Inference Network (GPIN), which is an evolutionary graph learning method to purify the visual entities and capture the core semantics. It is observed that the dense distribution of reduplicative objects and the crowd of semantically dependent OCR tokens usually co-exist in the image. Motivated by this, GPIN adopts an adaptive node dropping strategy to dynamically downscale semantically closed nodes for graph evolution and update. To deepen the comprehension of scene text, GPIN is a dual-path hierarchical graph architecture that progressively aggregates the evolved object graph and the evolved token graph semantics into a graph vector that serves as visual cues to facilitate the answer reasoning. It can effectively eliminate object redundancy and enhance the association of semantically continuous tokens. Experiments conducted on TextVQA and ST-VQA datasets show that GPIN achieves promising performance compared with state-of-the-art methods.

Förderung

  • National Key Research and Development Program of China
  • National Natural Science Foundation of China
  • Major Project of Anhui Province
  • University Synergy Innovation Program of Anhui Province

Zitieren

APA

Zhou, S., Guo, D., Yang, X., Dong, J., & Wang, M. (2024). Graph Pooling Inference Network for Text-based VQA. ACM Transactions on Multimedia Computing, Communications, and Applications, 20(4), 1-21. https://doi.org/10.1145/3634918

Vancouver

Zhou S, Guo D, Yang X, Dong J, Wang M. Graph Pooling Inference Network for Text-based VQA. ACM Transactions on Multimedia Computing, Communications, and Applications. 2024;20(4):1-21. doi:10.1145/3634918

BibTeX
@article{Zhou2024,
  author = {Zhou, Sheng and Guo, Dan and Yang, Xun and Dong, Jianfeng and Wang, Meng},
  title = {Graph Pooling Inference Network for Text-based VQA},
  journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
  year = {2024},
  volume = {20},
  number = {4},
  pages = {1--21},
  doi = {10.1145/3634918}
}
Kostenlos starten →