Deep-Learning Enhanced Augmented Reality for Robotic Radical Prostatectomy on Consumer Grade Hardware
Surgical Innovation · 06.11.2026
DOI: 10.1177/15533506261504154 ↗Kurz gefasst
Fachartikel in der Zeitschrift Surgical Innovation, online erschienen am 06.11.2026. Verfasst von Vincent Benard und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Urology, University Hospitals of Geneva, Geneva, Switzerland. Die Arbeit nennt 47 Literaturangaben. Gefördert von: Foundation Nicod.
Abstract (Originalsprache)
Background
Augmented Reality (AR) during Robotic-Assisted Radical Prostatectomy (RARP) provides surgical guidance by overlaying patient-specific 3D Virtual Models (VM) onto the intraoperative view. A main drawback is instrument occlusion, where VM overlaps the surgical tools, disturbing the augmented view. Deep learning (DL)-based desocclusion addresses this issue, but current implementation rely on expensive, specialized hardware.
Methods
We developed an AR desocclusion pipeline running on consumer-grade hardware (∼1150 US Dollars). For model selection, we trained and benchmarked 21 encoder architectures paired with a U-Net decoder. The selected model was integrated into a macOS application using CoreML inference on the Apple Neural Engine. Memory usage, thermal state and framerate were monitored on a continuous 81-minute console session. Photon-to-photon latency (from surgical field to the surgeon eye) was measured on the DaVinci® Xi using a microcontroller-phototransistor setup.
Results
FastViT-S12/Unet offered the highest accuracy among models meeting our inference-time budget (∼2 ms inference), achieving a Dice score of 95.1% (95% CI, 94.3-95.7) internally, 91.6% (95% CI, 90.2-92.9) and 90.2% (95% CI, 84.7-94.4) on externals tests, with a mean inference time of 1.89 ± 0.36 ms. The pipeline maintained 57.98 frames per second using 1.47 GB of unified memory with a nominal thermal state. On the surgical console, photon-to-photon median latency increased from 60 ms (Q1-Q3: 57.5-62.5) to 120.5 ms (Q1-Q3: 111-131) with the AR pipeline.
Conclusions
Real-time AR-RARP with DL-based desocclusion is achievable on low-cost hardware with an added latency remaining within ranges reported as technically acceptable for intraoperative use.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Urology, University Hospitals of Geneva, Geneva, Switzerland.
- Division of Medical Information Sciences, University Hospitals of Geneva, Geneva, Switzerland.
- Department of Radiology and Medical Informatics, University of Geneva, Geneva, Switzerland.
- Departement of Surgery, University Hospitals of Geneva, Geneva, Switzerland.
ORCID-Profile: Vincent Benard, Olivier Windisch
Förderung
- Foundation Nicod
Zitieren
APA
Benard, V., Windisch, O., Bjelogrlic, M., Turbé, H., Lovis, C., Diana, M., & Valerio, M. (2026). Deep-Learning Enhanced Augmented Reality for Robotic Radical Prostatectomy on Consumer Grade Hardware. Surgical Innovation. https://doi.org/10.1177/15533506261504154
Vancouver
Benard V, Windisch O, Bjelogrlic M, Turbé H, Lovis C, Diana M, et al. Deep-Learning Enhanced Augmented Reality for Robotic Radical Prostatectomy on Consumer Grade Hardware. Surgical Innovation. 2026. doi:10.1177/15533506261504154
BibTeX
@article{Benard2026,
author = {Benard, Vincent and Windisch, Olivier and Bjelogrlic, Mina and Turbé, Hugues and Lovis, Christian and Diana, Michele and Valerio, Massimo},
title = {Deep-Learning Enhanced Augmented Reality for Robotic Radical Prostatectomy on Consumer Grade Hardware},
journal = {Surgical Innovation},
year = {2026},
doi = {10.1177/15533506261504154}
}
EUVIMED