HIST: Hierarchical and sequential transformer for image captioning
IET Computer Vision · 2024-10 · 18 (7) · 1043-1056
DOI: 10.1049/cvi2.12305 ↗Kurz gefasst
Fachartikel in der Zeitschrift IET Computer Vision, online erschienen am 15.08.2024. Verfasst von Feixiao Lv und 3 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 75 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).
Abstract (Originalsprache)
Image captioning aims to automatically generate a natural language description of a given image, and most state‐of‐the‐art models have adopted an encoder–decoder transformer framework. Such transformer structures, however, show two main limitations in the task of image captioning. Firstly, the traditional transformer obtains high‐level fusion features to decode while ignoring other‐level features, resulting in losses of image content. Secondly, the transformer is weak in modelling the natural order characteristics of language. To address theseissues, the authors propose a HI erarchical and S equential T ransformer ( HIST ) structure, which forces each layer of the encoder and decoder to focus on features of different granularities, and strengthen the sequentially semantic information. Specifically, to capture the details of different levels of features in the image, the authors combine the visual features of multiple regions and divide them into multiple levels differently. In addition, to enhance the sequential information, the sequential enhancement module in each decoder layer block extracts different levels of features for sequentially semantic extraction and expression. Extensive experiments on the public datasets MS‐COCO and Flickr30k have demonstrated the effectiveness of our proposed method, and show that the authors’ method outperforms most of previous state of the arts.
Zitieren
APA
Lv, F., Wang, R., Jing, L., & Dai, P. (2024). HIST: Hierarchical and sequential transformer for image captioning. IET Computer Vision, 18(7), 1043-1056. https://doi.org/10.1049/cvi2.12305
Vancouver
Lv F, Wang R, Jing L, Dai P. HIST: Hierarchical and sequential transformer for image captioning. IET Computer Vision. 2024;18(7):1043-1056. doi:10.1049/cvi2.12305
BibTeX
@article{Lv2024,
author = {Lv, Feixiao and Wang, Rui and Jing, Lihua and Dai, Pengwen},
title = {HIST: Hierarchical and sequential transformer for image captioning},
journal = {IET Computer Vision},
year = {2024},
volume = {18},
number = {7},
pages = {1043--1056},
doi = {10.1049/cvi2.12305}
}
EUVIMED