EUVIMEDEUVIMED entdecken
Crossref

Prototype-Based Semantic Segmentation

Tianfei Zhou, Wenguan Wang

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2024-10 · 46 (10) · 6858-6872

DOI: 10.1109/tpami.2024.3387116 ↗

Kurz gefasst

Fachartikel in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024-10. Verfasst von Tianfei Zhou und 1 weiteren Autor:innen. Die Arbeit wurde bisher 82-mal zitiert (Crossref) und nennt 166 Literaturangaben. Gefördert von: National Natural Science Foundation of China, CCF-Tencent Open Fund.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Deep learning based semantic segmentation solutions have yielded compelling results over the preceding decade. They encompass diverse network architectures (FCN based or attention based), along with various mask decoding schemes (parametric softmax based or pixel-query based). Despite the divergence, they can be grouped within a unified framework by interpreting the softmax weights or query vectors as learnable class prototypes. In light of this prototype view, we reveal inherent limitations within the parametric segmentation regime, and accordingly develop a nonparametric alternative based on non-learnable prototypes. In contrast to previous approaches that entail the learning of a single weight/query vector per class in a fully parametric manner, our approach represents each class as a set of non-learnable prototypes, relying solely upon the mean features of training pixels within that class. The pixel-wise prediction is thus achieved by nonparametric nearest prototype retrieving. This allows our model to directly shape the pixel embedding space by optimizing the arrangement between embedded pixels and anchored prototypes. It is able to accommodate an arbitrary number of classes with a constant number of learnable parameters. Through empirical evaluation with FCN based and Transformer based segmentation models (i.e., HRNet, Swin, SegFormer, Mask2Former) and backbones (i.e., ResNet, HRNet, Swin, MiT), our nonparametric framework shows superior performance on standard segmentation datasets (i.e., ADE20 K, Cityscapes, COCO-Stuff), as well as in large-vocabulary semantic segmentation scenarios. We expect that this study will provoke a rethink of the current de facto semantic segmentation model design.

Förderung

  • National Natural Science Foundation of China
  • CCF-Tencent Open Fund

Zitieren

APA

Zhou, T., & Wang, W. (2024). Prototype-Based Semantic Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(10), 6858-6872. https://doi.org/10.1109/tpami.2024.3387116

Vancouver

Zhou T, Wang W. Prototype-Based Semantic Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024;46(10):6858-6872. doi:10.1109/tpami.2024.3387116

BibTeX
@article{Zhou2024,
  author = {Zhou, Tianfei and Wang, Wenguan},
  title = {Prototype-Based Semantic Segmentation},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2024},
  volume = {46},
  number = {10},
  pages = {6858--6872},
  doi = {10.1109/tpami.2024.3387116}
}
Kostenlos starten →