Markerless Tracking of Robotic Surgical Instruments With Head Mounted Display for Augmented Reality Applications
Healthcare Technology Letters · 2025-01 · 12 (1)
DOI: 10.1049/htl2.70044 ↗Kurz gefasst
Fachartikel in der Zeitschrift Healthcare Technology Letters, online erschienen am 16.12.2025. Verfasst von Nicholas Greene und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science Johns Hopkins University Baltimore Maryland USA. Die Arbeit nennt 25 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
In robotic‐assisted laparoscopic surgery, an assistant surgeon stands at the bedside assisting the intervention, while the surgeon sits at the console teleoperating the robot. Tasks for the assistant include navigating new instruments into the surgeon's field‐of‐view and passing in or retracting materials from the body using hand‐held tools. We previously developed ARssist , an augmented reality application based on an optical see‐through head‐mounted display (HMD), to aid the assistant. Localization of the HMD with respect to the robot was achieved via the attachment of markers. In this paper, we propose a novel markerless tracking method for robotic instruments using a HoloLens 2 HMD. We first run off‐the‐shelf YOLOv11 and SAMURAI (an adaptation of Segment Anything 2) networks to detect instrument primitives (shaft lines and keypoints). We then recover full 6D poses via a geometrically interpretable pipeline combining perspective‐n‐point (PnP) and a multi‐view least‐squares optimization. We experimentally compare the markerless tracking accuracy to a baseline marker‐based tracking solution, and show similar instrument tip accuracy. This suggests that the markerless method is an acceptable substitute to marker‐based tracking for this augmented reality application, while avoiding workflow issues with sterilizing and attaching markers.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Computer Science Johns Hopkins University Baltimore Maryland USA.
- Department of Computer Science and Engineering The Chinese University of Hong Kong HKSAR China.
ORCID-Profile: Peter Kazanzides
Zitieren
APA
Greene, N., Long, A., Long, Y., Han, Z., Dou, Q., & Kazanzides, P. (2025). Markerless Tracking of Robotic Surgical Instruments With Head Mounted Display for Augmented Reality Applications. Healthcare Technology Letters, 12(1). https://doi.org/10.1049/htl2.70044
Vancouver
Greene N, Long A, Long Y, Han Z, Dou Q, Kazanzides P. Markerless Tracking of Robotic Surgical Instruments With Head Mounted Display for Augmented Reality Applications. Healthcare Technology Letters. 2025;12(1). doi:10.1049/htl2.70044
BibTeX
@article{Greene2025,
author = {Greene, Nicholas and Long, Aoqi and Long, Yonghao and Han, Zheng and Dou, Qi and Kazanzides, Peter},
title = {Markerless Tracking of Robotic Surgical Instruments With Head Mounted Display for Augmented Reality Applications},
journal = {Healthcare Technology Letters},
year = {2025},
volume = {12},
number = {1},
doi = {10.1049/htl2.70044}
}
EUVIMED