EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

A Fusion Encoder with Multi-Task Guidance for Cross-Modal Text–Image Retrieval in Remote Sensing

Xiong Zhang, Weipeng Li, Xu Wang, Luyao Wang, Fuzhong Zheng, Long Wang, Haisu Zhang

Remote Sensing · 21.09.2023 · 15 (18) · 4637

DOI: 10.3390/rs15184637 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Remote Sensing, online erschienen am 21.09.2023. Verfasst von Xiong Zhang und 6 weiteren Autor:innen. Die Arbeit wurde bisher 19-mal zitiert (Crossref) und nennt 53 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In recent years, there has been a growing interest in remote sensing image–text cross-modal retrieval due to the rapid development of space information technology and the significant increase in the volume of remote sensing image data. Remote sensing images have unique characteristics that make the cross-modal retrieval task challenging. Firstly, the semantics of remote sensing images are fine-grained, meaning they can be divided into multiple basic units of semantic expression. Different combinations of basic units of semantic expression can generate diverse text descriptions. Additionally, these images exhibit variations in resolution, color, and perspective. To address these challenges, this paper proposes a multi-task guided fusion encoder (MTGFE) based on the multimodal fusion encoding method, the progressiveness of which has been proved in the cross-modal retrieval of natural images. By jointly training the model with three tasks: image–text matching (ITM), masked language modeling (MLM), and the newly introduced multi-view joint representations contrast (MVJRC), we enhance its capability to capture fine-grained correlations between remote sensing images and texts. Specifically, the MVJRC task is designed to improve the model’s consistency in joint representation expression and fine-grained correlation, particularly for remote sensing images with significant differences in resolution, color, and angle. Furthermore, to address the computational complexity associated with large-scale fusion models and improve retrieval efficiency, this paper proposes a retrieval filtering method, which achieves higher retrieval efficiency while minimizing accuracy loss. Extensive experiments were conducted on four public datasets to evaluate the proposed method, and the results validate its effectiveness.

Förderung

  • National Natural Science Foundation of China

Zitieren

APA

Zhang, X., Li, W., Wang, X., Wang, L., Zheng, F., Wang, L., & Zhang, H. (2023). A Fusion Encoder with Multi-Task Guidance for Cross-Modal Text–Image Retrieval in Remote Sensing. Remote Sensing, 15(18), 4637. https://doi.org/10.3390/rs15184637

Vancouver

Zhang X, Li W, Wang X, Wang L, Zheng F, Wang L, et al. A Fusion Encoder with Multi-Task Guidance for Cross-Modal Text–Image Retrieval in Remote Sensing. Remote Sensing. 2023;15(18):4637. doi:10.3390/rs15184637

BibTeX
@article{Zhang2023,
  author = {Zhang, Xiong and Li, Weipeng and Wang, Xu and Wang, Luyao and Zheng, Fuzhong and Wang, Long and Zhang, Haisu},
  title = {A Fusion Encoder with Multi-Task Guidance for Cross-Modal Text–Image Retrieval in Remote Sensing},
  journal = {Remote Sensing},
  year = {2023},
  volume = {15},
  number = {18},
  pages = {4637},
  doi = {10.3390/rs15184637}
}
Kostenlos starten →