A Fusion Encoder with Multi-Task Guidance for Cross-Modal Text–Image Retrieval in Remote Sensing
Remote Sensing · 21.09.2023 · 15 (18) · 4637
DOI: 10.3390/rs15184637 ↗Kurz gefasst
Fachartikel in der Zeitschrift Remote Sensing, online erschienen am 21.09.2023. Verfasst von Xiong Zhang und 6 weiteren Autor:innen. Die Arbeit wurde bisher 19-mal zitiert (Crossref) und nennt 53 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China.
Abstract (Originalsprache)
In recent years, there has been a growing interest in remote sensing image–text cross-modal retrieval due to the rapid development of space information technology and the significant increase in the volume of remote sensing image data. Remote sensing images have unique characteristics that make the cross-modal retrieval task challenging. Firstly, the semantics of remote sensing images are fine-grained, meaning they can be divided into multiple basic units of semantic expression. Different combinations of basic units of semantic expression can generate diverse text descriptions. Additionally, these images exhibit variations in resolution, color, and perspective. To address these challenges, this paper proposes a multi-task guided fusion encoder (MTGFE) based on the multimodal fusion encoding method, the progressiveness of which has been proved in the cross-modal retrieval of natural images. By jointly training the model with three tasks: image–text matching (ITM), masked language modeling (MLM), and the newly introduced multi-view joint representations contrast (MVJRC), we enhance its capability to capture fine-grained correlations between remote sensing images and texts. Specifically, the MVJRC task is designed to improve the model’s consistency in joint representation expression and fine-grained correlation, particularly for remote sensing images with significant differences in resolution, color, and angle. Furthermore, to address the computational complexity associated with large-scale fusion models and improve retrieval efficiency, this paper proposes a retrieval filtering method, which achieves higher retrieval efficiency while minimizing accuracy loss. Extensive experiments were conducted on four public datasets to evaluate the proposed method, and the results validate its effectiveness.
Förderung
- National Natural Science Foundation of China
Zitieren
APA
Zhang, X., Li, W., Wang, X., Wang, L., Zheng, F., Wang, L., & Zhang, H. (2023). A Fusion Encoder with Multi-Task Guidance for Cross-Modal Text–Image Retrieval in Remote Sensing. Remote Sensing, 15(18), 4637. https://doi.org/10.3390/rs15184637
Vancouver
Zhang X, Li W, Wang X, Wang L, Zheng F, Wang L, et al. A Fusion Encoder with Multi-Task Guidance for Cross-Modal Text–Image Retrieval in Remote Sensing. Remote Sensing. 2023;15(18):4637. doi:10.3390/rs15184637
BibTeX
@article{Zhang2023,
author = {Zhang, Xiong and Li, Weipeng and Wang, Xu and Wang, Luyao and Zheng, Fuzhong and Wang, Long and Zhang, Haisu},
title = {A Fusion Encoder with Multi-Task Guidance for Cross-Modal Text–Image Retrieval in Remote Sensing},
journal = {Remote Sensing},
year = {2023},
volume = {15},
number = {18},
pages = {4637},
doi = {10.3390/rs15184637}
}
EUVIMED