EUVIMEDEUVIMED entdecken
Crossref

Learning Semantic-aware Representation in Visual-Language Models for Multi-label Recognition with Partial Labels

Haoxian Ruan, Zhihua Xu, Zhijing Yang, Yongyi Lu, Jinghui Qin, Tianshui Chen

ACM Transactions on Multimedia Computing, Communications, and Applications · 31.03.2025 · 21 (3) · 1-19

DOI: 10.1145/3708991 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 19.02.2025. Verfasst von Haoxian Ruan und 5 weiteren Autor:innen. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 56 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Multi-label recognition with partial labels (MLR-PL), in which only some labels are known while others are unknown for each image, is a practical task in computer vision, since collecting large-scale and complete multi-label datasets is difficult in real application scenarios. Recently, vision language models (e.g., CLIP) have demonstrated impressive transferability to downstream tasks in data limited or label limited settings. However, current CLIP-based methods suffer from semantic confusion in MLR task due to the lack of fine-grained information in the single global visual and textual representation for all categories. In this work, we address this problem by introducing a semantic decoupling module and a category-specific prompt optimization method in CLIP-based framework. Specifically, the semantic decoupling module following the visual encoder learns category-specific feature maps by utilizing the semantic-guided spatial attention mechanism. Moreover, the category-specific prompt optimization method is introduced to learn text representations aligned with category semantics. Therefore, the prediction of each category is independent, which alleviate the semantic confusion problem. Extensive experiments on Microsoft COCO 2014 and Pascal VOC 2007 datasets demonstrate that the proposed framework significantly outperforms current state-of-art methods with a simpler model structure. Additionally, visual analysis shows that our method effectively separates information from different categories and achieves better performance compared to CLIP-based baseline method.

Zitieren

APA

Ruan, H., Xu, Z., Yang, Z., Lu, Y., Qin, J., & Chen, T. (2025). Learning Semantic-aware Representation in Visual-Language Models for Multi-label Recognition with Partial Labels. ACM Transactions on Multimedia Computing, Communications, and Applications, 21(3), 1-19. https://doi.org/10.1145/3708991

Vancouver

Ruan H, Xu Z, Yang Z, Lu Y, Qin J, Chen T. Learning Semantic-aware Representation in Visual-Language Models for Multi-label Recognition with Partial Labels. ACM Transactions on Multimedia Computing, Communications, and Applications. 2025;21(3):1-19. doi:10.1145/3708991

BibTeX
@article{Ruan2025,
  author = {Ruan, Haoxian and Xu, Zhihua and Yang, Zhijing and Lu, Yongyi and Qin, Jinghui and Chen, Tianshui},
  title = {Learning Semantic-aware Representation in Visual-Language Models for Multi-label Recognition with Partial Labels},
  journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
  year = {2025},
  volume = {21},
  number = {3},
  pages = {1--19},
  doi = {10.1145/3708991}
}
Kostenlos starten →