EUVIMEDScopri EUVIMED
Open access · CC BY 4.0

Integration of the latent variable knowledge into deep image captioning with Bayesian modeling

Alireza Barati, Hassan Farsi, Sajad Mohamadzadeh

IET Image Processing · 2023-05 · 17 (7) · 2256-2271

DOI: 10.1049/ipr2.12790 ↗

At a glance

Journal article in IET Image Processing, published online on 2023-03-25. Written by Alireza Barati and 2 co-authors. Cited 3 times so far (Crossref); 51 references. The full text is freely available (open access, CC BY 4.0).

AbstractDati bibliograficiCiteRelated studies

Abstract (original language)

Automatic image captioning systems assign one or more sentences to images to describe their visual content. Most of these systems use attention‐based deep convolutional neural networks and recurrent neural networks (CNN‐RNN‐Att). However, they must optimally use latent variables and side information within the image concepts. This study aims to integrate a latent variable into image captioning using CNN‐RNN‐Att. A Bayesian modeling framework is used for this work. As an instance of a latent variable, High‐Level Semantic Concepts (HLSCs) of tags are used to implement the proposed model. The Bayesian model output interpretation is to localize the entire image description process and breaks it down into sub‐problems. Thus, a baseline descriptor subnet is trained independently for each sub‐problem, and it is the only expert in captioning for a given HLSC. The final output is the caption derived from the subnet; its HLSC is closest to the image content. The results indicate that CNN‐RNN‐Att applied to data localized using HLSCs improves the captioning accuracy of the proposed method, which can be compared to the latest state‐of‐the‐art and most accurate captioning systems.

Cite

APA

Barati, A., Farsi, H., & Mohamadzadeh, S. (2023). Integration of the latent variable knowledge into deep image captioning with Bayesian modeling. IET Image Processing, 17(7), 2256-2271. https://doi.org/10.1049/ipr2.12790

Vancouver

Barati A, Farsi H, Mohamadzadeh S. Integration of the latent variable knowledge into deep image captioning with Bayesian modeling. IET Image Processing. 2023;17(7):2256-2271. doi:10.1049/ipr2.12790

BibTeX
@article{Barati2023,
  author = {Barati, Alireza and Farsi, Hassan and Mohamadzadeh, Sajad},
  title = {Integration of the latent variable knowledge into deep image captioning with Bayesian modeling},
  journal = {IET Image Processing},
  year = {2023},
  volume = {17},
  number = {7},
  pages = {2256--2271},
  doi = {10.1049/ipr2.12790}
}
Start for free →