Relation-Aware Multi-Pass Comparison Deconfounded Network for Change Captioning
IEEE Transactions on Circuits and Systems for Video Technology · 2024-12 · 34 (12) · 13349-13363
DOI: 10.1109/tcsvt.2024.3445337 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems for Video Technology, 2024-12. Verfasst von Zhicong Lu und 8 weiteren Autor:innen. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 59 Literaturangaben. Gefördert von: National Natural Science Foundation of China.
Abstract (Originalsprache)
Change captioning aims to describe the semantic change between a pair of images with natural language while remaining immune to viewpoint change. Based on the encoder-decoder architecture, most existing methods primarily focus on encoding effective change representations for transmission to the decoder. However, they suffer from an insufficient understanding of visual semantics, inadequate single-pass feature comparison, and a confounding bias caused by imbalanced viewpoint change data. These impair change representations and hinder unbiased caption generation. In this paper, we analyze and identify the confounding bias from a causality perspective and propose a Relation-aware Multi-pass Comparison Deconfounded (RMCD) network for change captioning, which elevates the encoding of change representations and mitigates the bias. Specifically, in the encoding stage, to sufficiently understand visual semantics, a position-guided context aggregating module is presented to capture the positional and contextual relations among objects in the image. Then, to achieve comprehensive change representations, we present a multi-pass feature comparison module to recognize semantic differences at various feature levels and progressively integrate them. In the decoding stage, to generate de-biased captions, the causal intervention is employed to remove the confounding bias which introduces spurious correlations between encoded change representations and captions. The newly achieved state-of-the-art performance on four publicly available benchmark datasets and further visual analysis demonstrate the superiority of our method.
Förderung
- National Natural Science Foundation of China
Zitieren
APA
Lu, Z., Jin, L., Chen, Z., Tian, C., Sun, X., Li, X., Zhang, Y., Li, Q., & Xu, G. (2024). Relation-Aware Multi-Pass Comparison Deconfounded Network for Change Captioning. IEEE Transactions on Circuits and Systems for Video Technology, 34(12), 13349-13363. https://doi.org/10.1109/tcsvt.2024.3445337
Vancouver
Lu Z, Jin L, Chen Z, Tian C, Sun X, Li X, et al. Relation-Aware Multi-Pass Comparison Deconfounded Network for Change Captioning. IEEE Transactions on Circuits and Systems for Video Technology. 2024;34(12):13349-13363. doi:10.1109/tcsvt.2024.3445337
BibTeX
@article{Lu2024,
author = {Lu, Zhicong and Jin, Li and Chen, Ziwei and Tian, Changyuan and Sun, Xian and Li, Xiaoyu and Zhang, Yi and Li, Qi and Xu, Guangluan},
title = {Relation-Aware Multi-Pass Comparison Deconfounded Network for Change Captioning},
journal = {IEEE Transactions on Circuits and Systems for Video Technology},
year = {2024},
volume = {34},
number = {12},
pages = {13349--13363},
doi = {10.1109/tcsvt.2024.3445337}
}
EUVIMED