EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Novel Advance Image Caption Generation Utilizing Vision Transformer and Generative Adversarial Networks

Shourya Tyagi, Olukayode Ayodele Oki, Vineet Verma, Swati Gupta, Meenu Vijarania, Joseph Bamidele Awotunde, Abdulrauph Olanrewaju Babatunde

Computers · 22.11.2024 · 13 (12) · 305

DOI: 10.3390/computers13120305 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Computers, online erschienen am 22.11.2024. Verfasst von Shourya Tyagi und 6 weiteren Autor:innen. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 38 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Walter Sisulu University, East London, South Africa 5200 Komani Campus, South Africa.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In this paper, we propose a novel method for producing image captions through the utilization of Generative Adversarial Networks (GANs) and Vision Transformers (ViTs) using our proposed Image Captioning Utilizing Transformer and GAN (ICTGAN) model. Here we use the efficient representation learning of the ViTs to improve the realistic image production of the GAN. Using textual features from the LSTM-based language model, our proposed model combines salient information extracted from images using ViTs. This merging of features is made possible using a self-attention mechanism, which enables the model to efficiently take in and process data from both textual and visual sources using the self-attention properties of the self-attention mechanism. We perform various tests on the MS COCO dataset as well as the Flickr30k dataset, which are popular benchmark datasets for image-captioning tasks, to verify the effectiveness of our proposed model. The outcomes represent that, on this dataset, our algorithm outperforms other approaches in terms of relevance, diversity, and caption quality. With this, our model is robust to changes in the content and style of the images, demonstrating its excellent generalization skills. We also explain the benefits of our method, which include better visual–textual alignment, better caption coherence, and better handling of complicated scenarios. All things considered, our work represents a significant step forward in the field of picture caption creation, offering a complete solution that leverages the complementary advantages of GANs and ViT-based self-attention models. This work pushes the limits of what is currently possible in image caption generation, creating a new standard in the industry.

Förderung

  • Walter Sisulu University, East London, South Africa 5200 Komani Campus, South Africa

Zitieren

APA

Tyagi, S., Oki, O. A., Verma, V., Gupta, S., Vijarania, M., Awotunde, J. B., & Babatunde, A. O. (2024). Novel Advance Image Caption Generation Utilizing Vision Transformer and Generative Adversarial Networks. Computers, 13(12), 305. https://doi.org/10.3390/computers13120305

Vancouver

Tyagi S, Oki OA, Verma V, Gupta S, Vijarania M, Awotunde JB, et al. Novel Advance Image Caption Generation Utilizing Vision Transformer and Generative Adversarial Networks. Computers. 2024;13(12):305. doi:10.3390/computers13120305

BibTeX
@article{Tyagi2024,
  author = {Tyagi, Shourya and Oki, Olukayode Ayodele and Verma, Vineet and Gupta, Swati and Vijarania, Meenu and Awotunde, Joseph Bamidele and Babatunde, Abdulrauph Olanrewaju},
  title = {Novel Advance Image Caption Generation Utilizing Vision Transformer and Generative Adversarial Networks},
  journal = {Computers},
  year = {2024},
  volume = {13},
  number = {12},
  pages = {305},
  doi = {10.3390/computers13120305}
}
Kostenlos starten →