EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Vision and Language Reference for a Segment Anything Model for Few-Shot Segmentation

Kosuke Sakurai iD, Ryotaro Shimizu iD, Masayuki Goto iD

Journal of Imaging · 24.03.2026 · 12 (4) · 143

DOI: 10.3390/jimaging12040143 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Imaging, online erschienen am 24.03.2026. Verfasst von Kosuke Sakurai und 2 weiteren Autor:innen. Beteiligte Einrichtung: Graduate School of Creative Science and Engineering, Waseda University, Tokyo 169-8555, Japan. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 67 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Japan Society for the Promotion of Science, JSPS KAKENHI.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Segment Anything Model (SAM)-based few-shot segmentation models traditionally rely solely on annotated reference images as prompts, which inherently limits their accuracy due to an over-reliance on visual cues and a lack of semantic context. This reliance leads to incorrect segmentation, where visually similar objects from different categories are incorrectly identified as the target object. We propose Vision and Language Reference Prompt into SAM (VLP-SAM), a novel few-shot segmentation model that integrates both visual information of reference images and semantic information of text labels into SAM. VLP-SAM introduces a vision-language model (VLM) with pixel–text matching into the prompt encoder for SAM, effectively leveraging textual semantic consistency while preserving SAM’s extensive segmentation knowledge. By incorporating task-specific structures such as an attention mask, our model achieves superior few-shot segmentation performance with only 1.4 M learnable parameters. Evaluations on PASCAL-5i and COCO-20i datasets demonstrate that VLP-SAM significantly outperforms previous methods by 6.8% and 9.3% in mIoU, respectively. Furthermore, VLP-SAM exhibits strong generalization across unseen objects and cross-domain scenarios, highlighting the robustness provided by textual semantic guidance. This study offers an effective and scalable framework for few-shot segmentation with multimodal prompts.

Schlagwörter

Stichwörter der Autor:innen

few-shot segmentationimage segmentationmultimodal learningsegment anything modelvision-language model

Autor:innen und Institutionen

  • Graduate School of Creative Science and Engineering, Waseda University, Tokyo 169-8555, Japan.
  • Institute of Data Science, Waseda University, Tokyo 169-8555, Japan.
  • School of Creative Science and Engineering, Waseda University, Tokyo 169-8555, Japan.

ORCID-Profile: Kosuke Sakurai, Ryotaro Shimizu, Masayuki Goto

Förderung

  • Japan Society for the Promotion of Science
  • JSPS KAKENHI

Zitieren

APA

Sakurai, K., Shimizu, R., & Goto, M. (2026). Vision and Language Reference for a Segment Anything Model for Few-Shot Segmentation. Journal of Imaging, 12(4), 143. https://doi.org/10.3390/jimaging12040143

Vancouver

Sakurai K, Shimizu R, Goto M. Vision and Language Reference for a Segment Anything Model for Few-Shot Segmentation. Journal of Imaging. 2026;12(4):143. doi:10.3390/jimaging12040143

BibTeX
@article{Sakurai2026,
  author = {Sakurai, Kosuke and Shimizu, Ryotaro and Goto, Masayuki},
  title = {Vision and Language Reference for a Segment Anything Model for Few-Shot Segmentation},
  journal = {Journal of Imaging},
  year = {2026},
  volume = {12},
  number = {4},
  pages = {143},
  doi = {10.3390/jimaging12040143}
}
Kostenlos starten →