Insights into Object Semantics: Leveraging Transformer Networks for Advanced Image Captioning
Sensors · 11.03.2024 · 24 (6) · 1796
DOI: 10.3390/s24061796 ↗Kurz gefasst
Fachartikel in der Zeitschrift Sensors, online erschienen am 11.03.2024. Verfasst von Deema Abdal Hafeth und 1 weiteren Autor:innen. Beteiligte Einrichtung: School of Computer Science, University of Lincoln, Lincoln LN6 7TS, UK. Die Arbeit wurde bisher 16-mal zitiert (Crossref) und nennt 58 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Image captioning is a technique used to generate descriptive captions for images. Typically, it involves employing a Convolutional Neural Network (CNN) as the encoder to extract visual features, and a decoder model, often based on Recurrent Neural Networks (RNNs), to generate the captions. Recently, the encoder–decoder architecture has witnessed the widespread adoption of the self-attention mechanism. However, this approach faces certain challenges that require further research. One such challenge is that the extracted visual features do not fully exploit the available image information, primarily due to the absence of semantic concepts. This limitation restricts the ability to fully comprehend the content depicted in the image. To address this issue, we present a new image-Transformer-based model boosted with image object semantic representation. Our model incorporates semantic representation in encoder attention, enhancing visual features by integrating instance-level concepts. Additionally, we employ Transformer as the decoder in the language generation module. By doing so, we achieve improved performance in generating accurate and diverse captions. We evaluated the performance of our model on the MS-COCO and novel MACE datasets. The results illustrate that our model aligns with state-of-the-art approaches in terms of caption generation.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- School of Computer Science, University of Lincoln, Lincoln LN6 7TS, UK.
- School of Electrical & Computer Engineering, National Technical University of Athens, 15780 Athens, Greece.
ORCID-Profile: Stefanos Kollias
Zitieren
APA
Abdal Hafeth, D., & Kollias, S. (2024). Insights into Object Semantics: Leveraging Transformer Networks for Advanced Image Captioning. Sensors, 24(6), 1796. https://doi.org/10.3390/s24061796
Vancouver
Abdal Hafeth D, Kollias S. Insights into Object Semantics: Leveraging Transformer Networks for Advanced Image Captioning. Sensors. 2024;24(6):1796. doi:10.3390/s24061796
BibTeX
@article{AbdalHafeth2024,
author = {Abdal Hafeth, Deema and Kollias, Stefanos},
title = {Insights into Object Semantics: Leveraging Transformer Networks for Advanced Image Captioning},
journal = {Sensors},
year = {2024},
volume = {24},
number = {6},
pages = {1796},
doi = {10.3390/s24061796}
}
EUVIMED