Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perception
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026 · 1-18
DOI: 10.1109/tpami.2026.3748834 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026. Verfasst von Junjie Wang und 6 weiteren Autor:innen.
Abstract (Originalsprache)
Dense visual perception tasks have been constrained by their reliance on predefined categories, limiting their applicability in real-world scenarios where visual concepts are unbounded. While Vision-Language Models (VLMs) like CLIP have shown promise in open-vocabulary tasks, their direct application to dense perception often leads to suboptimal performance due to limitations in local feature representation. In this work, we present our observation that CLIP's image tokens struggle to effectively aggregate information from spatially or semantically related regions, resulting in features that lack local discriminability and spatial consistency. To address this issue, we propose DeCLIP, a novel framework that enhances CLIP by decoupling the self-attention module to obtain "content" and "context" features respectively. The context features are enhanced by jointly distilling semantic correlations from Vision Foundation Models (VFMs) and object integrity cues from diffusion models, thereby enhancing spatial consistency. In parallel, the content features are aligned with image crop representations and constrained by region correlations from VFMs to improve local discriminability. Extensive experiments demonstrate that DeCLIP establishes a solid foundation for open-vocabulary dense perception, consistently achieving state-of-the-art performance across a broad spectrum of tasks, including 2D detection and segmentation, 3D instance segmentation, video instance segmentation, and 6D object pose estimation.
Zitieren
APA
Wang, J., Chen, K., Li, Y., Chen, B., Zhao, H., Qi, X., & Tian, Z. (2026). Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perception. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1-18. https://doi.org/10.1109/tpami.2026.3748834
Vancouver
Wang J, Chen K, Li Y, Chen B, Zhao H, Qi X, et al. Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perception. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2026:1-18. doi:10.1109/tpami.2026.3748834
BibTeX
@article{Wang2026,
author = {Wang, Junjie and Chen, Keyu and Li, Yulin and Chen, Bin and Zhao, Hengshuang and Qi, Xiaojuan and Tian, Zhuotao},
title = {Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perception},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2026},
pages = {1--18},
doi = {10.1109/tpami.2026.3748834}
}
EUVIMED