ISTR: Mask-Embedding-Based Instance Segmentation Transformer
IEEE Transactions on Image Processing · 2024 · 33 · 2895-2907
DOI: 10.1109/tip.2024.3385980 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Image Processing, 2024. Verfasst von Jie Hu und 3 weiteren Autor:innen. Die Arbeit wurde bisher 23-mal zitiert (Crossref) und nennt 66 Literaturangaben. Gefördert von: National Key Research and Development Program of China, National Science Fund for Distinguished Young Scholars, National Natural Science Foundation of China.
Abstract (Originalsprache)
Transformer-based instance-level recognition has attracted increasing research attention recently due to the superior performance. However, although attempts have been made to encode masks as embeddings into Transformer-based frameworks, how to combine mask embeddings and spatial information for a transformer-based approach is still not fully explored. In this paper, we revisit the design of mask-embedding-based pipelines and propose an Instance Segmentation TRansformer (ISTR) with Mask Meta-Embeddings (MME), leveraging the strengths of transformer models in encoding embedding information and incorporating spatial information from mask embeddings. ISTR incorporates a recurrent refining head that consists of a Dynamic Box Predictor (DBP), a Mask Information Generator (MIG), and a Mask Meta-Decoder (MMD). To improve the quality of mask embeddings, MME interprets the mask encoding-decoding processes as a mutual information maximization problem, which unifies the objective functions of different decoding schemes such as Principal Component Analysis (PCA) and Discrete Cosine Transform (DCT) with a meta-formulation. Under the meta-formulation, a learnable Spatial Mask Tuner (SMT) is further proposed, which fuses the spatial and embedding information produced from MIG and can significantly boost the segmentation performance. The resulting varieties, i.e., ISTR-PCA, ISTR-DCT, and ISTR-SMT, demonstrate the effectiveness and efficiency of incorporating mask embeddings with the query-based instance segmentation pipelines. On the COCO dataset, ISTR surpasses all predominant mask-embedding-based models by a large margin, and achieves competitive performance compared to concurrent state-of-the-art models. On the Cityscapes dataset, ISTR also outperforms several strong baselines. Our code has been made available at: https://github.com/hujiecpp/ISTR.
Förderung
- National Key Research and Development Program of China
- National Science Fund for Distinguished Young Scholars
- National Natural Science Foundation of China
- Natural Science Foundation of Fujian Province of China
Zitieren
APA
Hu, J., Lu, Y., Zhang, S., & Cao, L. (2024). ISTR: Mask-Embedding-Based Instance Segmentation Transformer. IEEE Transactions on Image Processing, 33, 2895-2907. https://doi.org/10.1109/tip.2024.3385980
Vancouver
Hu J, Lu Y, Zhang S, Cao L. ISTR: Mask-Embedding-Based Instance Segmentation Transformer. IEEE Transactions on Image Processing. 2024;33:2895-2907. doi:10.1109/tip.2024.3385980
BibTeX
@article{Hu2024,
author = {Hu, Jie and Lu, Yao and Zhang, Shengchuan and Cao, Liujuan},
title = {ISTR: Mask-Embedding-Based Instance Segmentation Transformer},
journal = {IEEE Transactions on Image Processing},
year = {2024},
volume = {33},
pages = {2895--2907},
doi = {10.1109/tip.2024.3385980}
}
EUVIMED