A transformer-based Urdu image caption generation
Journal of Ambient Intelligence and Humanized Computing · 2024-09 · 15 (9) · 3441-3457
DOI: 10.1007/s12652-024-04824-9 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Ambient Intelligence and Humanized Computing, online erschienen am 02.07.2024. Verfasst von Muhammad Hadi und 7 weiteren Autor:innen. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 55 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Image caption generation has emerged as a remarkable development that bridges the gap between Natural Language Processing (NLP) and Computer Vision (CV). It lies at the intersection of these fields and presents unique challenges, particularly when dealing with low-resource languages such as Urdu. Limited research on basic Urdu language understanding necessitates further exploration in this domain. In this study, we propose three Seq2Seq-based architectures specifically tailored for Urdu image caption generation. Our approach involves leveraging transformer models to generate captions in Urdu, a significantly more challenging task than English. To facilitate the training and evaluation of our models, we created an Urdu-translated subset of the flickr8k dataset, which contains images featuring dogs in action accompanied by corresponding Urdu captions. Our designed models encompassed a deep learning-based approach, utilizing three different architectures: Convolutional Neural Network (CNN) + Long Short-term Memory (LSTM) with Soft attention employing word2Vec embeddings, CNN+Transformer, and Vit+Roberta models. Experimental results demonstrate that our proposed model outperforms existing state-of-the-art approaches, achieving 86 BLEU-1 and 90 BERT-F1 scores. The generated Urdu image captions exhibit syntactic, contextual, and semantic correctness. Our study highlights the inherent challenges associated with retraining models on low-resource languages. Our findings highlight the potential of pre-trained models for facilitating the development of NLP and CV applications in low-resource language settings.
Zitieren
APA
Hadi, M., Safder, I., Waheed, H., Zaman, F., Aljohani, N. R., Nawaz, R., Hassan, S. U., & Sarwar, R. (2024). A transformer-based Urdu image caption generation. Journal of Ambient Intelligence and Humanized Computing, 15(9), 3441-3457. https://doi.org/10.1007/s12652-024-04824-9
Vancouver
Hadi M, Safder I, Waheed H, Zaman F, Aljohani NR, Nawaz R, et al. A transformer-based Urdu image caption generation. Journal of Ambient Intelligence and Humanized Computing. 2024;15(9):3441-3457. doi:10.1007/s12652-024-04824-9
BibTeX
@article{Hadi2024,
author = {Hadi, Muhammad and Safder, Iqra and Waheed, Hajra and Zaman, Farooq and Aljohani, Naif Radi and Nawaz, Raheel and Hassan, Saeed Ul and Sarwar, Raheem},
title = {A transformer-based Urdu image caption generation},
journal = {Journal of Ambient Intelligence and Humanized Computing},
year = {2024},
volume = {15},
number = {9},
pages = {3441--3457},
doi = {10.1007/s12652-024-04824-9}
}
EUVIMED