EUVIMEDEUVIMED entdecken
Crossref

Active Supervised Cross-Modal Retrieval

Huaiwen Zhang, Yang Yang, Fan Qi, Shengsheng Qian, Changsheng Xu

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2025-06 · 47 (6) · 5112-5126

DOI: 10.1109/tpami.2025.3550526 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025-06. Verfasst von Huaiwen Zhang und 4 weiteren Autor:innen. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 69 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Program for Young Talents of Science and Technology in Universities of Inner Mongolia Autonomous Region.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Supervised Cross-Modal Retrieval (SCMR) achieves significant performance with the supervision provided by substantial label annotations of multi-modal data. However, the requirement for large annotated multi-modal datasets restricts the use of supervised cross-modal retrieval in many practical scenarios. Active Learning (AL) has been proposed to reduce labeling costs while improving performance in various label-dependent tasks, in which the most informative unlabeled samples are selected for labeling and training. Directly exploiting the existing AL methods for supervised cross-modal retrieval may not be a good idea since they only focus on the uncertainty within each modality, ignoring the inter-modality relationship within the text-image pairs. Furthermore, existing methods focus exclusively on the informativeness of data during sample selection, leading to a biased, homogenized set where selected samples often contain nearly identical semantics and are densely distributed in a region of the feature space. Persistent training with such biased data selections can disturb multi-modal representation learning and substantially degrade the retrieval performance of SCMR. In this work, we propose an Active Supervised Cross-Modal Retrieval (ASCMR) framework, which effectively identifies informative multi-modal samples and generates unbiased sample selections. In particular, we propose a probabilistic multi-modal informativeness estimation that captures both the intra-modality and inter-modality uncertainty of multi-modal pairs within a unified representation. To ensure unbiased sample selection, we introduce a density-aware budget allocation strategy that constrains the active learning objective of maximizing the informativeness of selection with a novel semantic density regularization term. The proposed methods are evaluated on three widely used benchmark datasets, MS-COCO, NUS-WIDE, and MIRFlickr, demonstrating our effectiveness in significantly reducing the annotation cost while outperforming other baselines of active learning strategies. We could achieve over 95% of the fully supervised model's performance by only utilizing 6%, 3%, and 4% active selected samples for MS-COCO, NUS-WIDE, and MIRFlickr, respectively.

Förderung

  • National Natural Science Foundation of China
  • Program for Young Talents of Science and Technology in Universities of Inner Mongolia Autonomous Region

Zitieren

APA

Zhang, H., Yang, Y., Qi, F., Qian, S., & Xu, C. (2025). Active Supervised Cross-Modal Retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(6), 5112-5126. https://doi.org/10.1109/tpami.2025.3550526

Vancouver

Zhang H, Yang Y, Qi F, Qian S, Xu C. Active Supervised Cross-Modal Retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2025;47(6):5112-5126. doi:10.1109/tpami.2025.3550526

BibTeX
@article{Zhang2025,
  author = {Zhang, Huaiwen and Yang, Yang and Qi, Fan and Qian, Shengsheng and Xu, Changsheng},
  title = {Active Supervised Cross-Modal Retrieval},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2025},
  volume = {47},
  number = {6},
  pages = {5112--5126},
  doi = {10.1109/tpami.2025.3550526}
}
Kostenlos starten →