A Simple Switchable Framework for Open-Vocabulary Video Instance Segmentation
ACM Transactions on Multimedia Computing, Communications, and Applications · 31.07.2026 · 22 (7) · 1-21
DOI: 10.1145/3803013 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Multimedia Computing, Communications, and Applications, online erschienen am 24.06.2026. Verfasst von Feng Zhu und 2 weiteren Autor:innen. Die Arbeit nennt 69 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: Beijing Natural Science FoundationBeijing Natural Science Foundation, National Natural Science Foundation of China.
Abstract (Originalsprache)
Recently, the challenging task of open-vocabulary video instance segmentation (OVVIS) has been proposed. The OVVIS task requires simultaneously classifying, segmenting, and tracking objects in videos from an open set of categories, including novel categories unseen during training. Previous approaches typically rely on universal object proposals, memory-induced tracking, and open-vocabulary classification, which are often incompatible with established VIS and open-vocabulary segmentation methods. Observing that recent VIS methods share a common architecture decomposed into a segmenter and a tracker, we design a simple yet effective switchable open-vocabulary VIS (SOV) framework. SOV consists of an open-vocabulary segmenter and a dual memory tracker. The segmenter incorporates a frozen CLIP vision encoder as the backbone to enhance generalization on novel categories. The dual memory tracker is training-free and utilizes a dual-memory mechanism to enhance tracking robustness. Moreover, we can easily switch to various trackers. Benefiting from this design, SOV can inherit advantages from state-of-the-art VIS methods. To further optimize training efficiency, we propose a progressive “Long-Image, Short-Video” training pipeline. This strategy decouples the training process into an extensive image-level pre-training phase followed by a rapid video-level adaptation phase, significantly accelerating convergence while effectively bridging the domain gap between static images and dynamic videos. Our method outperforms previous methods by large margins on various benchmarks while maintaining faster inference speeds. Specifically, SOV achieves 38.0 mAP on the LV-VIS validation set. It also achieves strong zero-shot performance on popular VIS datasets (YTVIS19 50.9 mAP, YTVIS21 45.2 mAP, and OVIS 23.1 mAP), comparable to fully supervised methods. To further validate the flexibility of our switchable architecture, we extend SOV with the state-of-the-art CTVIS tracker, which yields improved performance (51.3 mAP) on YTVIS19. Code is available in the Supplementary Material.
Förderung
- Beijing Natural Science FoundationBeijing Natural Science Foundation
- National Natural Science Foundation of China
Zitieren
APA
Zhu, F., Chen, L., & Wei, Y. (2026). A Simple Switchable Framework for Open-Vocabulary Video Instance Segmentation. ACM Transactions on Multimedia Computing, Communications, and Applications, 22(7), 1-21. https://doi.org/10.1145/3803013
Vancouver
Zhu F, Chen L, Wei Y. A Simple Switchable Framework for Open-Vocabulary Video Instance Segmentation. ACM Transactions on Multimedia Computing, Communications, and Applications. 2026;22(7):1-21. doi:10.1145/3803013
BibTeX
@article{Zhu2026,
author = {Zhu, Feng and Chen, Ling and Wei, Yunchao},
title = {A Simple Switchable Framework for Open-Vocabulary Video Instance Segmentation},
journal = {ACM Transactions on Multimedia Computing, Communications, and Applications},
year = {2026},
volume = {22},
number = {7},
pages = {1--21},
doi = {10.1145/3803013}
}
EUVIMED