EUVIMEDEUVIMED entdecken
Crossref

SMART: Syntax-Calibrated Multi-Aspect Relation Transformer for Change Captioning

Yunbin Tu, Liang Li, Li Su, Zheng-Jun Zha, Qingming Huang

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-07 · 46 (7) · 4926-4943

DOI: 10.1109/tpami.2024.3365104 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-07. Verfasst von Yunbin Tu und 4 weiteren Autor:innen. Die Arbeit wurde bisher 68-mal zitiert (Crossref) und nennt 55 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Fundamental Research Funds for the Central Universities, Key R&D Plan Project of Zhejiang Province.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Change captioning aims to describe the semantic change between two similar images. In this process, as the most typical distractor, viewpoint change leads to the pseudo changes about appearance and position of objects, thereby overwhelming the real change. Besides, since the visual signal of change appears in a local region with weak feature, it is difficult for the model to directly translate the learned change features into the sentence. In this paper, we propose a syntax-calibrated multi-aspect relation transformer to learn effective change features under different scenes, and build reliable cross-modal alignment between the change features and linguistic words during caption generation. Specifically, a multi-aspect relation learning network is designed to 1) explore the fine-grained changes under irrelevant distractors (e.g., viewpoint change) by embedding the relations of semantics and relative position into the features of each image; 2) learn two view-invariant image representations by strengthening their global contrastive alignment relation, so as to help capture a stable difference representation; 3) provide the model with the prior knowledge about whether and where the semantic change happened by measuring the relation between the representations of captured difference and the image pair. Through the above manner, the model can learn effective change features for caption generation. Further, we introduce the syntax knowledge of Part-of-Speech (POS) and devise a POS-based visual switch to calibrate the transformer decoder. The POS-based visual switch dynamically utilizes visual information during different word generation based on the POS of words. This enables the decoder to build reliable cross-modal alignment, so as to generate a high-level linguistic sentence about change. Extensive experiments show that the proposed method achieves the state-of-the-art performance on the three public datasets.

Förderung

  • National Natural Science Foundation of China
  • Fundamental Research Funds for the Central Universities
  • Key R&D Plan Project of Zhejiang Province
  • Youth Innovation Promotion Association of the Chinese Academy of Sciences

Zitieren

APA

Tu, Y., Li, L., Su, L., Zha, Z. J., & Huang, Q. (2024). SMART: Syntax-Calibrated Multi-Aspect Relation Transformer for Change Captioning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(7), 4926-4943. https://doi.org/10.1109/tpami.2024.3365104

Vancouver

Tu Y, Li L, Su L, Zha ZJ, Huang Q. SMART: Syntax-Calibrated Multi-Aspect Relation Transformer for Change Captioning. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(7):4926-4943. doi:10.1109/tpami.2024.3365104

BibTeX
@article{Tu2024,
  author = {Tu, Yunbin and Li, Liang and Su, Li and Zha, Zheng-Jun and Huang, Qingming},
  title = {SMART: Syntax-Calibrated Multi-Aspect Relation Transformer for Change Captioning},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2024},
  volume = {46},
  number = {7},
  pages = {4926--4943},
  doi = {10.1109/tpami.2024.3365104}
}
Kostenlos starten →