EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Real-time deep learning semantic segmentation during intra-operative surgery for 3D augmented reality assistance

Leonardo Tanzi, Pietro Piazzolla, Francesco Porpiglia, Enrico Vezzetti

International Journal of Computer Assisted Radiology and Surgery · 2021-09 · 16 (9) · 1435-1445

DOI: 10.1007/s11548-021-02432-y ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Computer Assisted Radiology and Surgery, online erschienen am 24.06.2021. Verfasst von Leonardo Tanzi und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Management, Production and Design Engineering, Polytechnic University of Turin, Turin, Italy. leonardo.tanzi@polito.it. Hauptthemen laut MEDLINE: Augmented Reality, Deep Learning. Die Arbeit wurde bisher 97-mal zitiert (Crossref) und nennt 35 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Politecnico di Torino.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Purpose

The current study aimed to propose a Deep Learning (DL) and Augmented Reality (AR) based solution for a in-vivo robot-assisted radical prostatectomy (RARP), to improve the precision of a published work from our group. We implemented a two-steps automatic system to align a 3D virtual ad-hoc model of a patient’s organ with its 2D endoscopic image, to assist surgeons during the procedure.

Methods

This approach was carried out using a Convolutional Neural Network (CNN) based structure for semantic segmentation and a subsequent elaboration of the obtained output, which produced the needed parameters for attaching the 3D model. We used a dataset obtained from 5 endoscopic videos ( A, B, C, D, E ), selected and tagged by our team’s specialists. We then evaluated the most performing couple of segmentation architecture and neural network and tested the overlay performances.

Results

U-Net stood out as the most effecting architectures for segmentation. ResNet and MobileNet obtained similar Intersection over Unit (IoU) results but MobileNet was able to elaborate almost twice operations per seconds. This segmentation technique outperformed the results from the former work, obtaining an average IoU for the catheter of 0.894 ( σ = 0.076) compared to 0.339 ( σ = 0.195). This modifications lead to an improvement also in the 3D overlay performances, in particular in the Euclidean Distance between the predicted and actual model’s anchor point, from 12.569 ( σ = 4.456) to 4.160 ( σ = 1.448) and in the Geodesic Distance between the predicted and actual model’s rotations, from 0.266 ( σ = 0.131) to 0.169 ( σ = 0.073).

Conclusion

This work is a further step through the adoption of DL and AR in the surgery domain. In future works, we will overcome the limits of this approach and finally improve every step of the surgical procedure.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Augmented RealityDeep LearningHumansImage Processing, Computer-AssistedMaleNeural Networks, ComputerSemantics

Stichwörter der Autor:innen

Deep learningIntra-operativeNeural networkSemantic segmentation

Autor:innen und Institutionen

  • Department of Management, Production and Design Engineering, Polytechnic University of Turin, Turin, Italy. leonardo.tanzi@polito.it.
  • Department of Management, Production and Design Engineering, Polytechnic University of Turin, Turin, Italy.
  • Division of Urology, Department of Oncology, School of Medicine, University of Turin, Turin, Italy.

Förderung

  • Politecnico di Torino

Zitieren

APA

Tanzi, L., Piazzolla, P., Porpiglia, F., & Vezzetti, E. (2021). Real-time deep learning semantic segmentation during intra-operative surgery for 3D augmented reality assistance. International Journal of Computer Assisted Radiology and Surgery, 16(9), 1435-1445. https://doi.org/10.1007/s11548-021-02432-y

Vancouver

Tanzi L, Piazzolla P, Porpiglia F, Vezzetti E. Real-time deep learning semantic segmentation during intra-operative surgery for 3D augmented reality assistance. International Journal of Computer Assisted Radiology and Surgery. 2021;16(9):1435-1445. doi:10.1007/s11548-021-02432-y

BibTeX
@article{Tanzi2021,
  author = {Tanzi, Leonardo and Piazzolla, Pietro and Porpiglia, Francesco and Vezzetti, Enrico},
  title = {Real-time deep learning semantic segmentation during intra-operative surgery for 3D augmented reality assistance},
  journal = {International Journal of Computer Assisted Radiology and Surgery},
  year = {2021},
  volume = {16},
  number = {9},
  pages = {1435--1445},
  doi = {10.1007/s11548-021-02432-y}
}
Kostenlos starten →