On Distinctive Image Captioning via Comparing and Reweighting
IEEE Transactions on Pattern Analysis and Machine Intelligence · 01.02.2023 · 45 (2) · 2088-2103
DOI: 10.1109/tpami.2022.3159811 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, erschienen am 01.02.2023. Verfasst von Jiuniu Wang und 3 weiteren Autor:innen. Die Arbeit wurde bisher 31-mal zitiert (Crossref) und nennt 68 Literaturangaben. Gefördert von: Research Grants Council of the Hong Kong Special Administrative Region, City University of Hong Kong Strategic Research.
Abstract (Originalsprache)
Recent image captioning models are achieving impressive results based on popular metrics, i.e., BLEU, CIDEr, and SPICE. However, focusing on the most popular metrics that only consider the overlap between the generated captions and human annotation could result in using common words and phrases, which lacks distinctiveness, i.e., many similar images have the same caption. In this paper, we aim to improve the distinctiveness of image captions via comparing and reweighting with a set of similar images. First, we propose a distinctiveness metric-between-set CIDEr (CIDErBtw) to evaluate the distinctiveness of a caption with respect to those of similar images. Our metric reveals that the human annotations of each image in the MSCOCO dataset are not equivalent based on distinctiveness; however, previous works normally treat the human annotations equally during training, which could be a reason for generating less distinctive captions. In contrast, we reweight each ground-truth caption according to its distinctiveness during training. We further integrate a long-tailed weight strategy to highlight the rare words that contain more information, and captions from the similar image set are sampled as negative examples to encourage the generated sentence to be unique. Finally, extensive experiments are conducted, showing that our proposed approach significantly improves both distinctiveness (as measured by CIDErBtw and retrieval metrics) and accuracy (e.g., as measured by CIDEr) for a wide variety of image captioning baselines. These results are further confirmed through a user study.
Förderung
- Research Grants Council of the Hong Kong Special Administrative Region
- City University of Hong Kong Strategic Research
Zitieren
APA
Wang, J., Xu, W., Wang, Q., & Chan, A. B. (2023). On Distinctive Image Captioning via Comparing and Reweighting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2), 2088-2103. https://doi.org/10.1109/tpami.2022.3159811
Vancouver
Wang J, Xu W, Wang Q, Chan AB. On Distinctive Image Captioning via Comparing and Reweighting. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2023;45(2):2088-2103. doi:10.1109/tpami.2022.3159811
BibTeX
@article{Wang2023,
author = {Wang, Jiuniu and Xu, Wenjia and Wang, Qingzhong and Chan, Antoni B},
title = {On Distinctive Image Captioning via Comparing and Reweighting},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2023},
volume = {45},
number = {2},
pages = {2088--2103},
doi = {10.1109/tpami.2022.3159811}
}
EUVIMED