EUVIMEDEUVIMED entdecken
Crossref

CLIP-VIS: Adapting CLIP for Open-Vocabulary Video Instance Segmentation

Wenqi Zhu, Jiale Cao, Jin Xie, Shuangming Yang, Yanwei Pang

IEEE Transactions on Circuits and Systems for Video Technology · 2025-02 · 35 (2) · 1098-1110

DOI: 10.1109/tcsvt.2024.3474698 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Circuits and Systems for Video Technology, 2025-02. Verfasst von Wenqi Zhu und 4 weiteren Autor:innen. Die Arbeit wurde bisher 14-mal zitiert (Crossref) und nennt 71 Literaturangaben. Gefördert von: National Key Research and Development Program of China, Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Open-vocabulary video instance segmentation strives to segment and track instances belonging to an open set of categories in a videos. The vision-language model Contrastive Language-Image Pre-training (CLIP) has shown robust zero-shot classification ability in image-level open-vocabulary tasks. In this paper, we propose a simple encoder-decoder network, called CLIP-VIS, to adapt CLIP for open-vocabulary video instance segmentation. Our CLIP-VIS adopts frozen CLIP and introduces three modules, including class-agnostic mask generation, temporal topK-enhanced matching, and weighted open-vocabulary classification. Given a set of initial queries, class-agnostic mask generation introduces a pixel decoder and a transformer decoder on CLIP pre-trained image encoder to predict query masks and corresponding object scores and mask IoU scores. Then, temporal topK-enhanced matching performs query matching across frames using the K mostly matched frames. Finally, weighted open-vocabulary classification first employs mask pooling to generate query visual features from CLIP pre-trained image encoder, and second performs weighted classification using object scores and mask IoU scores. Our CLIP-VIS does not require the annotations of instance categories and identities. The experiments are performed on various video instance segmentation datasets, which demonstrate the effectiveness of our proposed method, especially for novel categories. When using ConvNeXt-B as backbone, our CLIP-VIS achieves the AP and APn scores of 32.2% and 40.2% on the validation set of LV-VIS dataset, which outperforms OV2Seg by 11.1% and 23.9% respectively. We will release the source code and models at https://github.com/zwq456/CLIP-VIS.git.

Förderung

  • National Key Research and Development Program of China
  • Natural Science Foundation of China

Zitieren

APA

Zhu, W., Cao, J., Xie, J., Yang, S., & Pang, Y. (2025). CLIP-VIS: Adapting CLIP for Open-Vocabulary Video Instance Segmentation. IEEE Transactions on Circuits and Systems for Video Technology, 35(2), 1098-1110. https://doi.org/10.1109/tcsvt.2024.3474698

Vancouver

Zhu W, Cao J, Xie J, Yang S, Pang Y. CLIP-VIS: Adapting CLIP for Open-Vocabulary Video Instance Segmentation. IEEE Transactions on Circuits and Systems for Video Technology. 2025;35(2):1098-1110. doi:10.1109/tcsvt.2024.3474698

BibTeX
@article{Zhu2025,
  author = {Zhu, Wenqi and Cao, Jiale and Xie, Jin and Yang, Shuangming and Pang, Yanwei},
  title = {CLIP-VIS: Adapting CLIP for Open-Vocabulary Video Instance Segmentation},
  journal = {IEEE Transactions on Circuits and Systems for Video Technology},
  year = {2025},
  volume = {35},
  number = {2},
  pages = {1098--1110},
  doi = {10.1109/tcsvt.2024.3474698}
}
Kostenlos starten →