EUVIMEDEUVIMED entdecken
Crossref

Causal Prompts for Open-Vocabulary Video Instance Segmentation

Rongkun Zheng, Lu Qi, Xi Chen, Yi Wang, Kun Wang, Yu Qiao, Hengshuang Zhao

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026-07 · 48 (7) · 8429-8443

DOI: 10.1109/tpami.2026.3669976 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026-07. Verfasst von Rongkun Zheng und 6 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 80 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Hong Kong Research Grant Council General Research, National Key R&D Program of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Open-vocabulary Video Instance Segmentation addresses the challenging task of detecting, segmenting, and tracking objects in videos, including categories not encountered during training. However, existing approaches often overlook rich temporal cues from preceding frames, limiting their ability to leverage causal context for robust open-world generalization. To bridge this gap, we propose CPOVIS, a novel framework that introduces causal prompts-dynamically propagated visual and taxonomy prompts from historical frames-to enhance temporal reasoning and semantic consistency. Built upon a Mask2Former architecture with a CLIP backbone, CPOVIS integrates three core innovations: 1) PromptCLIP, which aligns cross-modal embeddings while preserving open-vocabulary capabilities; 2) a Visual Prompt Injector that propagates object-level features to maintain spatial-temporal coherence; and 3) a Taxonomy Prompt Infuser that leverages hierarchical semantic relationships to stabilize unseen category recognition. Furthermore, we introduce a contrastive learning strategy to disentangle object representations across frames and adapt the Segment Anything Model (SAM2) to boost open-vocabulary segmentation and tracking capacity in open-vocabulary video scenarios. Extensive experiments on seven challenging open- and closed-vocabulary video segmentation benchmarks demonstrate CPOVIS's state-of-the-art performance, outperforming existing methods by significant margins. Our findings highlight the critical role of causal prompt propagation in advancing video understanding in open-world scenarios.

Förderung

  • National Natural Science Foundation of China
  • Hong Kong Research Grant Council General Research
  • National Key R&D Program of China

Zitieren

APA

Zheng, R., Qi, L., Chen, X., Wang, Y., Wang, K., Qiao, Y., & Zhao, H. (2026). Causal Prompts for Open-Vocabulary Video Instance Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(7), 8429-8443. https://doi.org/10.1109/tpami.2026.3669976

Vancouver

Zheng R, Qi L, Chen X, Wang Y, Wang K, Qiao Y, et al. Causal Prompts for Open-Vocabulary Video Instance Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2026;48(7):8429-8443. doi:10.1109/tpami.2026.3669976

BibTeX
@article{Zheng2026,
  author = {Zheng, Rongkun and Qi, Lu and Chen, Xi and Wang, Yi and Wang, Kun and Qiao, Yu and Zhao, Hengshuang},
  title = {Causal Prompts for Open-Vocabulary Video Instance Segmentation},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2026},
  volume = {48},
  number = {7},
  pages = {8429--8443},
  doi = {10.1109/tpami.2026.3669976}
}
Kostenlos starten →