EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Thangka Image Captioning Based on Semantic Concept Prompt and Multimodal Feature Optimization

Wenjin Hu iD, Lang Qiao, Wendong Kang, Xinyue Shi

Journal of Imaging · 16.08.2023 · 9 (8) · 162

DOI: 10.3390/jimaging9080162 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Imaging, online erschienen am 16.08.2023. Verfasst von Wenjin Hu und 3 weiteren Autor:innen. Beteiligte Einrichtung: School of Mathematics and Computer Science, Northwest Minzu Univsersity, Lanzhou 730030, China. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 46 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, Fundamental Research Funds for the Central Universities, The Nature Science Foundation of China.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Thangka images exhibit a high level of diversity and richness, and the existing deep learning-based image captioning methods generate poor accuracy and richness of Chinese captions for Thangka images. To address this issue, this paper proposes a Semantic Concept Prompt and Multimodal Feature Optimization network (SCAMF-Net). The Semantic Concept Prompt (SCP) module is introduced in the text encoding stage to obtain more semantic information about the Thangka by introducing contextual prompts, thus enhancing the richness of the description content. The Multimodal Feature Optimization (MFO) module is proposed to optimize the correlation between Thangka images and text. This module enhances the correlation between the image features and text features of the Thangka through the Captioner and Filter to more accurately describe the visual concept features of the Thangka. The experimental results demonstrate that our proposed method outperforms baseline models on the Thangka dataset in terms of BLEU-4, METEOR, ROUGE, CIDEr, and SPICE by 8.7%, 7.9%, 8.2%, 76.6%, and 5.7%, respectively. Furthermore, this method also exhibits superior performance compared to the state-of-the-art methods on the public MSCOCO dataset.

Schlagwörter

Stichwörter der Autor:innen

Thangkadeep learningimage captioningknowledge distillationvisual concepts

Autor:innen und Institutionen

  • School of Mathematics and Computer Science, Northwest Minzu Univsersity, Lanzhou 730030, China.
  • Key Laboratory of China's Ethnic Languages and Information Technology of Ministry of Education, Northwest Minzu University, Lanzhou 730030, China.

ORCID-Profile: Wenjin Hu

Förderung

  • National Natural Science Foundation of China
  • Fundamental Research Funds for the Central Universities
  • The Nature Science Foundation of China
  • Innovative Research for the innovative research team of SEAC
  • Key Laboratory of China’s Ethnic Languages and Information Technology of Ministry of Education

Zitieren

APA

Hu, W., Qiao, L., Kang, W., & Shi, X. (2023). Thangka Image Captioning Based on Semantic Concept Prompt and Multimodal Feature Optimization. Journal of Imaging, 9(8), 162. https://doi.org/10.3390/jimaging9080162

Vancouver

Hu W, Qiao L, Kang W, Shi X. Thangka Image Captioning Based on Semantic Concept Prompt and Multimodal Feature Optimization. Journal of Imaging. 2023;9(8):162. doi:10.3390/jimaging9080162

BibTeX
@article{Hu2023,
  author = {Hu, Wenjin and Qiao, Lang and Kang, Wendong and Shi, Xinyue},
  title = {Thangka Image Captioning Based on Semantic Concept Prompt and Multimodal Feature Optimization},
  journal = {Journal of Imaging},
  year = {2023},
  volume = {9},
  number = {8},
  pages = {162},
  doi = {10.3390/jimaging9080162}
}
Kostenlos starten →