EUVIMEDScopri EUVIMED
Crossref

Scale-Semantic Joint Decoupling Network for Image-Text Retrieval in Remote Sensing

Chengyu Zheng, Ning Song, Ruoyu Zhang, Lei Huang, Zhiqiang Wei, Jie Nie

ACM Transactions on Multimedia Computing, Communications, and Applications · 2024-01-31 · 20 (1) · 1-20

DOI: 10.1145/3603628 ↗

At a glance

Journal article in ACM Transactions on Multimedia Computing, Communications, and Applications, published online on 2023-08-24. Written by Chengyu Zheng and 5 co-authors. Cited 22 times so far (Crossref); 48 references. Funded by: National Natural Science Foundation of China, Fundamental Research Funds for the Central Universities, National Key Research and Development Program of China.

AbstractDati bibliograficiCiteRelated studies

Abstract (original language)

Image-text retrieval in remote sensing aims to provide flexible information for data analysis and application. In recent years, state-of-the-art methods are dedicated to “scale decoupling” and “semantic decoupling” strategies to further enhance the capability of representation. However, these previous approaches focus on either the disentangling scale or semantics but ignore merging these two ideas in a union model, which extremely limits the performance of cross-modal retrieval models. To address these issues, we propose a novel Scale-Semantic Joint Decoupling Network (SSJDN) for remote sensing image-text retrieval. Specifically, we design the Bidirectional Scale Decoupling (BSD) module, which exploits Salience Extraction Map (SEM) and Salience Suppression Map (SSM) units to adaptively extract potential features and suppress cumbersome features at other scales in a bidirectional pattern to yield different scale clues. Besides, we design the Label-supervised Semantic Decoupling (LSD) module by leveraging the category semantic labels as prior knowledge to supervise images and texts probing significant semantic-related information. Finally, we design a Semantic-guided Triple Loss (STL), which adaptively generates a constant to adjust the loss function to improve the probability of matching the same semantic image and text and shorten the convergence time of the retrieval model. Our proposed SSJDN outperforms state-of-the-art approaches in numerical experiments conducted on four benchmark remote sensing datasets.

Funding

  • National Natural Science Foundation of China
  • Fundamental Research Funds for the Central Universities
  • National Key Research and Development Program of China

Cite

APA

Zheng, C., Song, N., Zhang, R., Huang, L., Wei, Z., & Nie, J. (2024). Scale-Semantic Joint Decoupling Network for Image-Text Retrieval in Remote Sensing. ACM Transactions on Multimedia Computing, Communications, and Applications, 20(1), 1-20. https://doi.org/10.1145/3603628

Vancouver

Zheng C, Song N, Zhang R, Huang L, Wei Z, Nie J. Scale-Semantic Joint Decoupling Network for Image-Text Retrieval in Remote Sensing. ACM Transactions on Multimedia Computing, Communications, and Applications. 2024;20(1):1-20. doi:10.1145/3603628

BibTeX
@article{Zheng2024,
  author = {Zheng, Chengyu and Song, Ning and Zhang, Ruoyu and Huang, Lei and Wei, Zhiqiang and Nie, Jie},
  title = {Scale-Semantic Joint Decoupling Network for Image-Text Retrieval in Remote Sensing},
  journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
  year = {2024},
  volume = {20},
  number = {1},
  pages = {1--20},
  doi = {10.1145/3603628}
}
Start for free →