EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Towards multimodal graph neural networks for surgical instrument anticipation

Lars Wagner, Dennis N Schneider, Leon Mayer, Alissa Jell, Carolin Müller, Alexander Lenz, Alois Knoll, Dirk Wilhelm

International Journal of Computer Assisted Radiology and Surgery · 10.07.2024 · 19 (10) · 1929-1937

DOI: 10.1007/s11548-024-03226-8 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Computer Assisted Radiology and Surgery, online erschienen am 10.07.2024. Verfasst von Lars Wagner und 7 weiteren Autor:innen. Beteiligte Einrichtung: Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Munich, Germany. lars.wagner@tum.de. Hauptthemen laut MEDLINE: Neural Networks, Computer. Die Arbeit wurde bisher 17-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Bayerische Staatsministerium für Wirtschaft, Landesentwicklung und Energie.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Purpose

Decision support systems and context-aware assistance in the operating room have emerged as the key clinical applications supporting surgeons in their daily work and are generally based on single modalities. The model- and knowledge-based integration of multimodal data as a basis for decision support systems that can dynamically adapt to the surgical workflow has not yet been established. Therefore, we propose a knowledge-enhanced method for fusing multimodal data for anticipation tasks.

Methods

We developed a holistic, multimodal graph-based approach combining imaging and non-imaging information in a knowledge graph representing the intraoperative scene of a surgery. Node and edge features of the knowledge graph are extracted from suitable data sources in the operating room using machine learning. A spatiotemporal graph neural network architecture subsequently allows for interpretation of relational and temporal patterns within the knowledge graph. We apply our approach to the downstream task of instrument anticipation while presenting a suitable modeling and evaluation strategy for this task.

Results

Our approach achieves an F1 score of 66.86% in terms of instrument anticipation, allowing for a seamless surgical workflow and adding a valuable impact for surgical decision support systems. A resting recall of 63.33% indicates the non-prematurity of the anticipations.

Conclusion

This work shows how multimodal data can be combined with the topological properties of an operating room in a graph-based approach. Our multimodal graph architecture serves as a basis for context-sensitive decision support systems in laparoscopic surgery considering a comprehensive intraoperative operating scene.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Neural Networks, ComputerHumansWorkflowOperating RoomsDecision Support Systems, ClinicalMachine LearningSurgery, Computer-Assisted

Stichwörter der Autor:innen

Graph neural networksSurgical data scienceSurgical instrument anticipationSurgical process modeling

Autor:innen und Institutionen

  • Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Munich, Germany. lars.wagner@tum.de.
  • Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Munich, Germany.
  • Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Department of Surgery, Munich, Germany.
  • Technical University of Munich, TUM School of Computation, Information and Technology, Chair of Robotics, Artificial Intelligence and Real-Time Systems, Munich, Germany.

Förderung

  • Bayerische Staatsministerium für Wirtschaft, Landesentwicklung und Energie

Zitieren

APA

Wagner, L., Schneider, D. N., Mayer, L., Jell, A., Müller, C., Lenz, A., Knoll, A., & Wilhelm, D. (2024). Towards multimodal graph neural networks for surgical instrument anticipation. International Journal of Computer Assisted Radiology and Surgery, 19(10), 1929-1937. https://doi.org/10.1007/s11548-024-03226-8

Vancouver

Wagner L, Schneider DN, Mayer L, Jell A, Müller C, Lenz A, et al. Towards multimodal graph neural networks for surgical instrument anticipation. International Journal of Computer Assisted Radiology and Surgery. 2024;19(10):1929-1937. doi:10.1007/s11548-024-03226-8

BibTeX
@article{Wagner2024,
  author = {Wagner, Lars and Schneider, Dennis N and Mayer, Leon and Jell, Alissa and Müller, Carolin and Lenz, Alexander and Knoll, Alois and Wilhelm, Dirk},
  title = {Towards multimodal graph neural networks for surgical instrument anticipation},
  journal = {International Journal of Computer Assisted Radiology and Surgery},
  year = {2024},
  volume = {19},
  number = {10},
  pages = {1929--1937},
  doi = {10.1007/s11548-024-03226-8}
}
Kostenlos starten →