EUVIMEDEUVIMED entdecken
Crossref

Ask Questions With Double Hints: Visual Question Generation With Answer-Awareness and Region-Reference

Kai Shen, Lingfei Wu, Siliang Tang, Fangli Xu, Bo Long, Yueting Zhuang, Jian Pei

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-12 · 46 (12) · 9648-9660

DOI: 10.1109/tpami.2024.3425222 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-12. Verfasst von Kai Shen und 6 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 43 Literaturangaben. Gefördert von: National Key Research and Development Program of China, Zhejiang NSF, Chinese Knowledge Center of Engineering Science and Technology.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The visual question generation (VQG) task aims to generate human-like questions from an image and potentially other side information (e.g., answer type). Previous works on VQG fall in two aspects: i) They suffer from one image to many questions mapping problem, which leads to the failure of generating referential and meaningful questions from an image. ii) They fail to model complex implicit relations among the visual objects in an image and also overlook potential interactions between the side information and image. To address these limitations, we first propose a novel learning paradigm to generate visual questions with answer-awareness and region-reference. Concretely, we aim to ask the right visual questions with Double Hints - textual answers and visual regions of interests, which could effectively mitigate the existing one-to-many mapping issue. Particularly, we develop a simple methodology to self-learn the visual hints without introducing any additional human annotations. Furthermore, to capture these sophisticated relationships, we propose a new double-hints guided Graph-to-Sequence learning framework, which first models them as a dynamic graph and learns the implicit topology end-to-end, and then utilizes a graph-to-sequence model to generate the questions with double hints. Experimental results demonstrate the priority of our proposed method.

Förderung

  • National Key Research and Development Program of China
  • Zhejiang NSF
  • Chinese Knowledge Center of Engineering Science and Technology

Zitieren

APA

Shen, K., Wu, L., Tang, S., Xu, F., Long, B., Zhuang, Y., & Pei, J. (2024). Ask Questions With Double Hints: Visual Question Generation With Answer-Awareness and Region-Reference. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12), 9648-9660. https://doi.org/10.1109/tpami.2024.3425222

Vancouver

Shen K, Wu L, Tang S, Xu F, Long B, Zhuang Y, et al. Ask Questions With Double Hints: Visual Question Generation With Answer-Awareness and Region-Reference. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(12):9648-9660. doi:10.1109/tpami.2024.3425222

BibTeX
@article{Shen2024,
  author = {Shen, Kai and Wu, Lingfei and Tang, Siliang and Xu, Fangli and Long, Bo and Zhuang, Yueting and Pei, Jian},
  title = {Ask Questions With Double Hints: Visual Question Generation With Answer-Awareness and Region-Reference},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2024},
  volume = {46},
  number = {12},
  pages = {9648--9660},
  doi = {10.1109/tpami.2024.3425222}
}
Kostenlos starten →