Causal Prompts for Open-Vocabulary Video Instance Segmentation
IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026-07 · 48 (7) · 8429-8443
DOI: 10.1109/tpami.2026.3669976 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026-07. Verfasst von Rongkun Zheng und 6 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 80 Literaturangaben. Gefördert von: National Natural Science Foundation of China, Hong Kong Research Grant Council General Research, National Key R&D Program of China.
Abstract (Originalsprache)
Open-vocabulary Video Instance Segmentation addresses the challenging task of detecting, segmenting, and tracking objects in videos, including categories not encountered during training. However, existing approaches often overlook rich temporal cues from preceding frames, limiting their ability to leverage causal context for robust open-world generalization. To bridge this gap, we propose CPOVIS, a novel framework that introduces causal prompts-dynamically propagated visual and taxonomy prompts from historical frames-to enhance temporal reasoning and semantic consistency. Built upon a Mask2Former architecture with a CLIP backbone, CPOVIS integrates three core innovations: 1) PromptCLIP, which aligns cross-modal embeddings while preserving open-vocabulary capabilities; 2) a Visual Prompt Injector that propagates object-level features to maintain spatial-temporal coherence; and 3) a Taxonomy Prompt Infuser that leverages hierarchical semantic relationships to stabilize unseen category recognition. Furthermore, we introduce a contrastive learning strategy to disentangle object representations across frames and adapt the Segment Anything Model (SAM2) to boost open-vocabulary segmentation and tracking capacity in open-vocabulary video scenarios. Extensive experiments on seven challenging open- and closed-vocabulary video segmentation benchmarks demonstrate CPOVIS's state-of-the-art performance, outperforming existing methods by significant margins. Our findings highlight the critical role of causal prompt propagation in advancing video understanding in open-world scenarios.
Förderung
- National Natural Science Foundation of China
- Hong Kong Research Grant Council General Research
- National Key R&D Program of China
Zitieren
APA
Zheng, R., Qi, L., Chen, X., Wang, Y., Wang, K., Qiao, Y., & Zhao, H. (2026). Causal Prompts for Open-Vocabulary Video Instance Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(7), 8429-8443. https://doi.org/10.1109/tpami.2026.3669976
Vancouver
Zheng R, Qi L, Chen X, Wang Y, Wang K, Qiao Y, et al. Causal Prompts for Open-Vocabulary Video Instance Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2026;48(7):8429-8443. doi:10.1109/tpami.2026.3669976
BibTeX
@article{Zheng2026,
author = {Zheng, Rongkun and Qi, Lu and Chen, Xi and Wang, Yi and Wang, Kun and Qiao, Yu and Zhao, Hengshuang},
title = {Causal Prompts for Open-Vocabulary Video Instance Segmentation},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
year = {2026},
volume = {48},
number = {7},
pages = {8429--8443},
doi = {10.1109/tpami.2026.3669976}
}
EUVIMED