EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Augmenting efficient real‐time surgical instrument segmentation in video with point tracking and Segment Anything

Zijian Wu iD, Adam Schmidt, Peter Kazanzides, Septimiu E Salcudean

Healthcare Technology Letters · 2025-01 · 12 (1)

DOI: 10.1049/htl2.12111 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Healthcare Technology Letters, online erschienen am 30.12.2024. Verfasst von Zijian Wu und 3 weiteren Autor:innen. Beteiligte Einrichtung: Robotics and Control Laboratory, Department of Electrical and Computer Engineering The University of British Columbia Vancouver Canada. Die Arbeit wurde bisher 8-mal zitiert (Crossref) und nennt 43 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Canada Foundation for Innovation.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The Segment Anything model (SAM) is a powerful vision foundation model that is revolutionizing the traditional paradigm of segmentation. Despite this, a reliance on prompting each frame and large computational cost limit its usage in robotically assisted surgery. Applications, such as augmented reality guidance, require little user intervention along with efficient inference to be usable clinically. This study addresses these limitations by adopting lightweight SAM variants to meet the efficiency requirement and employing fine‐tuning techniques to enhance their generalization in surgical scenes. Recent advancements in tracking any point have shown promising results in both accuracy and efficiency, particularly when points are occluded or leave the field of view. Inspired by this progress, a novel framework is presented that combines an online point tracker with a lightweight SAM model that is fine‐tuned for surgical instrument segmentation. Sparse points within the region of interest are tracked and used to prompt SAM throughout the video sequence, providing temporal consistency. The quantitative results surpass the state‐of‐the‐art semi‐supervised video object segmentation method XMem on the EndoVis 2015 dataset with 84.8 IoU and 91.0 Dice. The method achieves promising performance that is comparable to XMem and transformer‐based fully supervised segmentation methods on ex vivo UCL dVRK and in vivo CholecSeg8k datasets. In addition, the proposed method shows promising zero‐shot generalization ability on the label‐free STIR dataset. In terms of efficiency, the method was tested on a single GeForce RTX 4060/4090 GPU respectively, achieving an over 25/90 FPS inference speed. Code is available at: https://github.com/zijianwu1231/SIS‐PT‐SAM .

Schlagwörter

Stichwörter der Autor:innen

image segmentationmedical roboticsrobot visionsurgery

Autor:innen und Institutionen

  • Robotics and Control Laboratory, Department of Electrical and Computer Engineering The University of British Columbia Vancouver Canada.
  • Department of Computer Science Johns Hopkins University Baltimore Maryland USA.

ORCID-Profile: Zijian Wu

Förderung

  • Canada Foundation for Innovation

Zitieren

APA

Wu, Z., Schmidt, A., Kazanzides, P., & Salcudean, S. E. (2025). Augmenting efficient real‐time surgical instrument segmentation in video with point tracking and Segment Anything. Healthcare Technology Letters, 12(1). https://doi.org/10.1049/htl2.12111

Vancouver

Wu Z, Schmidt A, Kazanzides P, Salcudean SE. Augmenting efficient real‐time surgical instrument segmentation in video with point tracking and Segment Anything. Healthcare Technology Letters. 2025;12(1). doi:10.1049/htl2.12111

BibTeX
@article{Wu2025,
  author = {Wu, Zijian and Schmidt, Adam and Kazanzides, Peter and Salcudean, Septimiu E},
  title = {Augmenting efficient real‐time surgical instrument segmentation in video with point tracking and Segment Anything},
  journal = {Healthcare Technology Letters},
  year = {2025},
  volume = {12},
  number = {1},
  doi = {10.1049/htl2.12111}
}
Kostenlos starten →