EUVIMEDEUVIMED entdecken
Crossref

Explainable ViT and SCNN-LSTM Framework for Audio-Assisted Image Captioning

Nidhi B. Shah, Amit P. Ganatra

Journal of Innovative Image Processing · 15.07.2026 · 8 (3) · 978-999

DOI: 10.36548/jiip.2026.3.012 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Innovative Image Processing, online erschienen am 15.07.2026. Verfasst von Nidhi B. Shah und 1 weiteren Autor:innen. Die Arbeit nennt 25 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Assistive technologies are crucial for image captioning, a process that involves generating textual captions of visual scenes. Many of the current methods, however, lack semantic understanding, are computationally complex, and are not interpretable. Based on these challenges, in this paper, we present an Explainable Vision Transformer and Depthwise Separable CNN-LSTM (ViT-SCNN-LSTM) architecture to achieve accurate and explainable image caption generation. The framework is based on a pre-trained Vision Transformer (ViT-B/16) to extract rich visual features and a lightweight Depthwise Separable SCNN-LSTM decoder to generate descriptive captions without high computational complexity. Grad-CAM is integrated to get visual explanations by identifying image areas that affect the generation of captions. The audio-assisted module also provides an option to turn captions into speech, making it easier for visually impaired users to access the generated captions. This model was developed using Python and tested on the Flickr8k image corpus (8,000 images, 40,000 captions). To evaluate the generalisation capability, cross-dataset experiments were also performed with Flickr30k and MSCOCO 2017. Experimental results achieved ROUGE-1, ROUGE-L, ROUGE-S, BLEU-1, BLEU-2, BLEU-3, and BLEU-4 scores of 0.87, 0.86, 0.85, 0.8848, 0.8750, 0.8649, and 0.8547, respectively. The results show the proposed approach offers an efficient, interpretable, and user-friendly solution for automatic image understanding and caption generation.

Zitieren

APA

Shah, N. B., & Ganatra, A. P. (2026). Explainable ViT and SCNN-LSTM Framework for Audio-Assisted Image Captioning. Journal of Innovative Image Processing, 8(3), 978-999. https://doi.org/10.36548/jiip.2026.3.012

Vancouver

Shah NB, Ganatra AP. Explainable ViT and SCNN-LSTM Framework for Audio-Assisted Image Captioning. Journal of Innovative Image Processing. 2026;8(3):978-999. doi:10.36548/jiip.2026.3.012

BibTeX
@article{Shah2026,
  author = {Shah, Nidhi B. and Ganatra, Amit P.},
  title = {Explainable ViT and SCNN-LSTM Framework for Audio-Assisted Image Captioning},
  journal = {Journal of Innovative Image Processing},
  year = {2026},
  volume = {8},
  number = {3},
  pages = {978--999},
  doi = {10.36548/jiip.2026.3.012}
}
Kostenlos starten →