Multi-Grained Representation Aggregating Transformer with Gating Cycle for Change Captioning
ACM Transactions on Multimedia Computing, Communications, and Applications · 31.10.2024 · 20 (10) · 1-23
DOI: 10.1145/3660346 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 12.09.2024. Verfasst von Shengbin Yue und 4 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 63 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Yunnan High-tech Industry Development Project, Yunnan Key Research and Development Plan.
Abstract (Originalsprache)
Change captioning aims to describe the difference within an image pair in natural language, which combines visual comprehension and language generation. Although significant progress has been achieved, it remains a key challenge of perceiving the object change from different perspectives, especially the severe situation with drastic viewpoint change. In this article, we propose a novel full-attentive network, namely Multi-grained Representation Aggregating Transformer (MURAT), to distinguish the actual change from viewpoint change. Specifically, the Pair Encoder first captures similar semantics between pairwise objects in a multi-level manner, which are regarded as the semantic cues of distinguishing the irrelevant change. Next, a novel Multi-grained Representation Aggregator (MRA) is designed to construct the reliable difference representation by employing both coarse- and fine-grained semantic cues. Finally, the language decoder generates a description of the change based on the output of MRA. Besides, the Gating Cycle Mechanism is introduced to facilitate the semantic consistency between difference representation learning and language generation with a reverse manipulation, so as to bridge the semantic gap between change features and text features. Extensive experiments demonstrate that the proposed MURAT can greatly improve the ability to describe the actual change in the distraction of irrelevant change and achieves state-of-the-art performance on three benchmarks, CLEVR-Change, CLEVR-DC, and Spot-the-Diff.
Förderung
- National Natural Science Foundation of China
- Yunnan High-tech Industry Development Project
- Yunnan Key Research and Development Plan
- Reserve Talents for Academic and Technological Leaders in Yunnan Province
Zitieren
APA
Yue, S., Tu, Y., Li, L., Gao, S., & Yu, Z. (2024). Multi-Grained Representation Aggregating Transformer with Gating Cycle for Change Captioning. ACM Transactions on Multimedia Computing, Communications, and Applications, 20(10), 1-23. https://doi.org/10.1145/3660346
Vancouver
Yue S, Tu Y, Li L, Gao S, Yu Z. Multi-Grained Representation Aggregating Transformer with Gating Cycle for Change Captioning. ACM Transactions on Multimedia Computing, Communications, and Applications. 2024;20(10):1-23. doi:10.1145/3660346
BibTeX
@article{Yue2024,
author = {Yue, Shengbin and Tu, Yunbin and Li, Liang and Gao, Shengxiang and Yu, Zhengtao},
title = {Multi-Grained Representation Aggregating Transformer with Gating Cycle for Change Captioning},
journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
year = {2024},
volume = {20},
number = {10},
pages = {1--23},
doi = {10.1145/3660346}
}
EUVIMED