EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Cross-Modal Retrieval and Semantic Refinement for Remote Sensing Image Captioning

Zhengxin Li, Wenzhe Zhao, Xuanyi Du, Guangyao Zhou, Songlin Zhang

Remote Sensing · 03.01.2024 · 16 (1) · 196

DOI: 10.3390/rs16010196 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Remote Sensing, online erschienen am 03.01.2024. Verfasst von Zhengxin Li und 4 weiteren Autor:innen. Die Arbeit wurde bisher 21-mal zitiert (Crossref) und nennt 48 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Two-stage remote sensing image captioning (RSIC) methods have achieved promising results by incorporating additional pre-trained remote sensing tasks to extract supplementary information and improve caption quality. However, these methods face limitations in semantic comprehension, as pre-trained detectors/classifiers are constrained by predefined labels, leading to an oversight of the intricate and diverse details present in remote sensing images (RSIs). Additionally, the handling of auxiliary remote sensing tasks separately can introduce challenges in ensuring seamless integration and alignment with the captioning process. To address these problems, we propose a novel cross-modal retrieval and semantic refinement (CRSR) RSIC method. Specifically, we employ a cross-modal retrieval model to retrieve relevant sentences of each image. The words in these retrieved sentences are then considered as primary semantic information, providing valuable supplementary information for the captioning process. To further enhance the quality of the captions, we introduce a semantic refinement module that refines the primary semantic information, which helps to filter out misleading information and emphasize visually salient semantic information. A Transformer Mapper network is introduced to expand the representation of image features beyond the retrieved supplementary information with learnable queries. Both the refined semantic tokens and visual features are integrated and fed into a cross-modal decoder for caption generation. Through extensive experiments, we demonstrate the superiority of our CRSR method over existing state-of-the-art approaches on the RSICD, the UCM-Captions, and the Sydney-Captions datasets

Zitieren

APA

Li, Z., Zhao, W., Du, X., Zhou, G., & Zhang, S. (2024). Cross-Modal Retrieval and Semantic Refinement for Remote Sensing Image Captioning. Remote Sensing, 16(1), 196. https://doi.org/10.3390/rs16010196

Vancouver

Li Z, Zhao W, Du X, Zhou G, Zhang S. Cross-Modal Retrieval and Semantic Refinement for Remote Sensing Image Captioning. Remote Sensing. 2024;16(1):196. doi:10.3390/rs16010196

BibTeX
@article{Li2024,
  author = {Li, Zhengxin and Zhao, Wenzhe and Du, Xuanyi and Zhou, Guangyao and Zhang, Songlin},
  title = {Cross-Modal Retrieval and Semantic Refinement for Remote Sensing Image Captioning},
  journal = {Remote Sensing},
  year = {2024},
  volume = {16},
  number = {1},
  pages = {196},
  doi = {10.3390/rs16010196}
}
Kostenlos starten →