EUVIMEDEUVIMED entdecken
Crossref

Exploring Visual Relationships via Transformer-based Graphs for Enhanced Image Captioning

Jingyu Li, Zhendong Mao, Hao Li, Weidong Chen, Yongdong Zhang

ACM Transactions on Multimedia Computing, Communications, and Applications · 31.05.2024 · 20 (5) · 1-23

DOI: 10.1145/3638558 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 22.01.2024. Verfasst von Jingyu Li und 4 weiteren Autor:innen. Die Arbeit wurde bisher 24-mal zitiert (Crossref) und nennt 64 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Science Fund for Creative Research Groups.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Image captioning (IC), bringing vision to language, has drawn extensive attention. A crucial aspect of IC is the accurate depiction of visual relations among image objects. Visual relations encompass two primary facets: content relations and structural relations. Content relations, which comprise geometric positions content (i.e., distances and sizes) and semantic interactions content (i.e., actions and possessives), unveil the mutual correlations between objects. In contrast, structural relations pertain to the topological connectivity of object regions. Existing Transformer-based methods typically resort to geometric positions to enhance the visual relations, yet only using the shallow geometric content is unable to precisely cover actional content correlations and structural connection relations. In this article, we adopt a comprehensive perspective to examine the correlations between objects, incorporating both content relations (i.e., geometric and semantic relations) and structural relations, with the aim of generating plausible captions. To achieve this, first, we construct a geometric graph from bounding box features and a semantic graph from the scene graph parser to model the content relations. Innovatively, we construct a topology graph that amalgamates the sparsity characteristics of the geometric and semantic graphs, enabling the representation of image structural relations. Second, we propose a novel unified approach to enrich image relation representations by integrating semantic, geometric, and structural relations into self-attention. Finally, in the language decoding stage, we further leverage the semantic relation as prior knowledge to generate accurate words. Extensive experiments on MS-COCO dataset demonstrate the effectiveness of our model, with improvements of CIDEr from 128.6% to 136.6%. Codes have been released at https://github.com/CrossmodalGroup/ER-SAN/tree/main/VG-Cap .

Förderung

  • National Natural Science Foundation of China
  • Science Fund for Creative Research Groups

Zitieren

APA

Li, J., Mao, Z., Li, H., Chen, W., & Zhang, Y. (2024). Exploring Visual Relationships via Transformer-based Graphs for Enhanced Image Captioning. ACM Transactions on Multimedia Computing, Communications, and Applications, 20(5), 1-23. https://doi.org/10.1145/3638558

Vancouver

Li J, Mao Z, Li H, Chen W, Zhang Y. Exploring Visual Relationships via Transformer-based Graphs for Enhanced Image Captioning. ACM Transactions on Multimedia Computing, Communications, and Applications. 2024;20(5):1-23. doi:10.1145/3638558

BibTeX
@article{Li2024,
  author = {Li, Jingyu and Mao, Zhendong and Li, Hao and Chen, Weidong and Zhang, Yongdong},
  title = {Exploring Visual Relationships via Transformer-based Graphs for Enhanced Image Captioning},
  journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
  year = {2024},
  volume = {20},
  number = {5},
  pages = {1--23},
  doi = {10.1145/3638558}
}
Kostenlos starten →