EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Towards a zero-shot low-latency navigation for open surgery augmented reality applications

Michael Schwimmbeck, Serouj Khajarian, Christopher Auer, Thomas Wittenberg, Stefanie Remmele

International Journal of Computer Assisted Radiology and Surgery · 2025

DOI: 10.1007/s11548-025-03480-4 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Computer Assisted Radiology and Surgery, 2025. Verfasst von Michael Schwimmbeck und 4 weiteren Autor:innen. Beteiligte Einrichtung: Research Group Medical Technologies, University of Applied Sciences Landshut, Landshut, Germany. michael.schwimmbeck@haw-landshut.de. Hauptthemen laut MEDLINE: Augmented Reality, Surgery, Computer-Assisted, Liver, Hepatectomy. Die Arbeit nennt 20 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Hochschule für angewandte Wissenschaften Landshut.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Purpose

Augmented reality (AR) enhances surgical navigation by superimposing visible anatomical structures with three-dimensional virtual models using head-mounted displays (HMDs). In particular, interventions such as open liver surgery can benefit from AR navigation, as it aids in identifying and distinguishing tumors and risk structures. However, there is a lack of automatic and markerless methods that are robust against real-world challenges, such as partial occlusion and organ motion.

Methods

We introduce a novel multi-device approach for automatic live navigation in open liver surgery that enhances the visualization and interaction capabilities of a HoloLens 2 HMD through precise and reliable registration using an Intel RealSense RGB-D camera. The intraoperative RGB-D segmentation and the preoperative CT data are utilized to register a virtual liver model to the target anatomy. An AR-prompted Segment Anything Model (SAM) enables robust segmentation of the liver in situ without the need for additional training data. To mitigate algorithmic latency, Double Exponential Smoothing (DES) is applied to forecast registration results.

Results

We conducted a phantom study for open liver surgery, investigating various scenarios of liver motion, viewpoints, and occlusion. The mean registration errors (8.31 mm–18.78 mm TRE) are comparable to those reported in prior work, while our approach demonstrates high success rates even for high occlusion factors and strong motion. Using forecasting, we bypassed the algorithmic latency of 79.8 ms per frame, with median forecasting errors below 2 mms and 1.5 degrees between the quaternions.

Conclusion

To our knowledge, this is the first work to approach markerless in situ visualization by combining a multi-device method with forecasting and a foundation model for segmentation and tracking. This enables a more reliable and precise AR registration of surgical targets with low latency. Our approach can be applied to other surgical applications and AR hardware with minimal effort.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Augmented RealitySurgery, Computer-AssistedLiverHepatectomyHumansPhantoms, ImagingImaging, Three-DimensionalTomography, X-Ray ComputedAlgorithms

Stichwörter der Autor:innen

Augmented realityForecastingHoloLensSegment anything modelSurgical navigation

Autor:innen und Institutionen

  • Research Group Medical Technologies, University of Applied Sciences Landshut, Landshut, Germany. michael.schwimmbeck@haw-landshut.de.
  • Chair for Visual Computing, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany. michael.schwimmbeck@haw-landshut.de.
  • Research Group Medical Technologies, University of Applied Sciences Landshut, Landshut, Germany.
  • Intelligent Embedded Systems Lab, University of Freiburg, Freiburg im Breisgau, Germany.
  • Chair for Visual Computing, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany.
  • Fraunhofer Institute for Integrated Circuits IIS, Erlangen, Germany.

Förderung

  • Hochschule für angewandte Wissenschaften Landshut

Zitieren

APA

Schwimmbeck, M., Khajarian, S., Auer, C., Wittenberg, T., & Remmele, S. (2025). Towards a zero-shot low-latency navigation for open surgery augmented reality applications. International Journal of Computer Assisted Radiology and Surgery. https://doi.org/10.1007/s11548-025-03480-4

Vancouver

Schwimmbeck M, Khajarian S, Auer C, Wittenberg T, Remmele S. Towards a zero-shot low-latency navigation for open surgery augmented reality applications. International Journal of Computer Assisted Radiology and Surgery. 2025. doi:10.1007/s11548-025-03480-4

BibTeX
@article{Schwimmbeck2025,
  author = {Schwimmbeck, Michael and Khajarian, Serouj and Auer, Christopher and Wittenberg, Thomas and Remmele, Stefanie},
  title = {Towards a zero-shot low-latency navigation for open surgery augmented reality applications},
  journal = {International Journal of Computer Assisted Radiology and Surgery},
  year = {2025},
  doi = {10.1007/s11548-025-03480-4}
}
Kostenlos starten →