EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Exploring Zero-Shot Semantic Segmentation with No Supervision Leakage

Yiqi Wang, Yingjie Tian

Electronics · 15.08.2023 · 12 (16) · 3452

DOI: 10.3390/electronics12163452 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Electronics, online erschienen am 15.08.2023. Verfasst von Yiqi Wang und 1 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 43 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Natural Science Foundation of China.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Zero-shot semantic segmentation (ZS3), the process of classifying unseen classes without explicit training samples, poses a significant challenge. Despite notable progress made by pre-trained vision-language models, they have a problem of “supervision leakage” in the unseen classes due to their large-scale pre-trained data. For example, CLIP is trained on 400M image–text pairs that contain large label space categories. So, it is not convincing for real “zero-shot” learning in machine learning. This paper introduces SwinZS3, an innovative framework that explores the “no-supervision-leakage” zero-shot semantic segmentation with an image encoder that is not pre-trained on the seen classes. SwinZS3 integrates the strengths of both visual and semantic embeddings within a unified joint embedding space. This approach unifies a transformer-based image encoder with a language encoder. A distinguishing feature of SwinZS3 is the implementation of four specialized loss functions in the training progress: cross-entropy loss, semantic-consistency loss, regression loss, and pixel-text score loss. These functions guide the optimization process based on dense semantic prototypes derived from the language encoder, making the encoder adept at recognizing unseen classes during inference without retraining. We evaluated SwinZS3 with standard ZS3 benchmarks, including PASCAL VOC and PASCAL Context. The outcomes affirm the effectiveness of our method, marking a new milestone in “no-supervison-leakage” ZS3 task performance.

Förderung

  • Natural Science Foundation of China

Zitieren

APA

Wang, Y., & Tian, Y. (2023). Exploring Zero-Shot Semantic Segmentation with No Supervision Leakage. Electronics, 12(16), 3452. https://doi.org/10.3390/electronics12163452

Vancouver

Wang Y, Tian Y. Exploring Zero-Shot Semantic Segmentation with No Supervision Leakage. Electronics. 2023;12(16):3452. doi:10.3390/electronics12163452

BibTeX
@article{Wang2023,
  author = {Wang, Yiqi and Tian, Yingjie},
  title = {Exploring Zero-Shot Semantic Segmentation with No Supervision Leakage},
  journal = {Electronics},
  year = {2023},
  volume = {12},
  number = {16},
  pages = {3452},
  doi = {10.3390/electronics12163452}
}
Kostenlos starten →