Explainable ViT and SCNN-LSTM Framework for Audio-Assisted Image Captioning
Journal of Innovative Image Processing · 15.07.2026 · 8 (3) · 978-999
DOI: 10.36548/jiip.2026.3.012 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Innovative Image Processing, online erschienen am 15.07.2026. Verfasst von Nidhi B. Shah und 1 weiteren Autor:innen. Die Arbeit nennt 25 Literaturangaben.
Abstract (Originalsprache)
Assistive technologies are crucial for image captioning, a process that involves generating textual captions of visual scenes. Many of the current methods, however, lack semantic understanding, are computationally complex, and are not interpretable. Based on these challenges, in this paper, we present an Explainable Vision Transformer and Depthwise Separable CNN-LSTM (ViT-SCNN-LSTM) architecture to achieve accurate and explainable image caption generation. The framework is based on a pre-trained Vision Transformer (ViT-B/16) to extract rich visual features and a lightweight Depthwise Separable SCNN-LSTM decoder to generate descriptive captions without high computational complexity. Grad-CAM is integrated to get visual explanations by identifying image areas that affect the generation of captions. The audio-assisted module also provides an option to turn captions into speech, making it easier for visually impaired users to access the generated captions. This model was developed using Python and tested on the Flickr8k image corpus (8,000 images, 40,000 captions). To evaluate the generalisation capability, cross-dataset experiments were also performed with Flickr30k and MSCOCO 2017. Experimental results achieved ROUGE-1, ROUGE-L, ROUGE-S, BLEU-1, BLEU-2, BLEU-3, and BLEU-4 scores of 0.87, 0.86, 0.85, 0.8848, 0.8750, 0.8649, and 0.8547, respectively. The results show the proposed approach offers an efficient, interpretable, and user-friendly solution for automatic image understanding and caption generation.
Zitieren
APA
Shah, N. B., & Ganatra, A. P. (2026). Explainable ViT and SCNN-LSTM Framework for Audio-Assisted Image Captioning. Journal of Innovative Image Processing, 8(3), 978-999. https://doi.org/10.36548/jiip.2026.3.012
Vancouver
Shah NB, Ganatra AP. Explainable ViT and SCNN-LSTM Framework for Audio-Assisted Image Captioning. Journal of Innovative Image Processing. 2026;8(3):978-999. doi:10.36548/jiip.2026.3.012
BibTeX
@article{Shah2026,
author = {Shah, Nidhi B. and Ganatra, Amit P.},
title = {Explainable ViT and SCNN-LSTM Framework for Audio-Assisted Image Captioning},
journal = {Journal of Innovative Image Processing},
year = {2026},
volume = {8},
number = {3},
pages = {978--999},
doi = {10.36548/jiip.2026.3.012}
}
EUVIMED