EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Semantic-Enhanced Cross-Modal Fusion for Improved Unsupervised Image Captioning

Nan Xiang, Ling Chen, Leiyan Liang, Xingdi Rao, Zehao Gong

Electronics · 22.08.2023 · 12 (17) · 3549

DOI: 10.3390/electronics12173549 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Electronics, online erschienen am 22.08.2023. Verfasst von Nan Xiang und 4 weiteren Autor:innen. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 38 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Natural Science Foundation of Chongqing Province of China, China Postdoctoral Science Foundation, Science and Technology Research Project of Chongqing Education Commission.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Unsupervised image captioning often grapples with challenges such as image–text mismatches and modality gaps, resulting in suboptimal captions. This paper introduces a semantic-enhanced cross-modal fusion model (SCFM) to address these issues. The SCFM integrates three innovative components: a text semantic enhancement network (TSE-Net) for nuanced semantic representation; contrast learning for optimizing similarity measures between text and images; and enhanced visual selection decoding (EVSD) for precise captioning. Unlike existing methods that struggle with capturing accurate semantic relationships and flexibility across scenarios, the proposed model provides a robust solution for unbiased and diverse captioning. Through experimental evaluations on the MS COCO and Flickr30k datasets, SCFM demonstrates significant improvements over the benchmark model, enhancing the CIDEr and BLEU-4 metrics by 3.6% and 3.2%, respectively. Visualization analysis further reveals the model’s superiority in increasing variability between hidden features and its potential in cross-domain and stylized image captioning. The findings not only contribute to the advancement of image captioning techniques but also open avenues for future research. Further investigations will explore SCFM’s adaptability to other multimodal tasks and refine it for more intricate image–text relationships.

Förderung

  • Natural Science Foundation of Chongqing Province of China
  • China Postdoctoral Science Foundation
  • Science and Technology Research Project of Chongqing Education Commission

Zitieren

APA

Xiang, N., Chen, L., Liang, L., Rao, X., & Gong, Z. (2023). Semantic-Enhanced Cross-Modal Fusion for Improved Unsupervised Image Captioning. Electronics, 12(17), 3549. https://doi.org/10.3390/electronics12173549

Vancouver

Xiang N, Chen L, Liang L, Rao X, Gong Z. Semantic-Enhanced Cross-Modal Fusion for Improved Unsupervised Image Captioning. Electronics. 2023;12(17):3549. doi:10.3390/electronics12173549

BibTeX
@article{Xiang2023,
  author = {Xiang, Nan and Chen, Ling and Liang, Leiyan and Rao, Xingdi and Gong, Zehao},
  title = {Semantic-Enhanced Cross-Modal Fusion for Improved Unsupervised Image Captioning},
  journal = {Electronics},
  year = {2023},
  volume = {12},
  number = {17},
  pages = {3549},
  doi = {10.3390/electronics12173549}
}
Kostenlos starten →