Towards multimodal graph neural networks for surgical instrument anticipation
International Journal of Computer Assisted Radiology and Surgery · 10.07.2024 · 19 (10) · 1929-1937
DOI: 10.1007/s11548-024-03226-8 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Computer Assisted Radiology and Surgery, online erschienen am 10.07.2024. Verfasst von Lars Wagner und 7 weiteren Autor:innen. Beteiligte Einrichtung: Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Munich, Germany. lars.wagner@tum.de. Hauptthemen laut MEDLINE: Neural Networks, Computer. Die Arbeit wurde bisher 17-mal zitiert (Crossref) und nennt 32 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Bayerische Staatsministerium für Wirtschaft, Landesentwicklung und Energie.
Abstract (Originalsprache)
Purpose
Decision support systems and context-aware assistance in the operating room have emerged as the key clinical applications supporting surgeons in their daily work and are generally based on single modalities. The model- and knowledge-based integration of multimodal data as a basis for decision support systems that can dynamically adapt to the surgical workflow has not yet been established. Therefore, we propose a knowledge-enhanced method for fusing multimodal data for anticipation tasks.
Methods
We developed a holistic, multimodal graph-based approach combining imaging and non-imaging information in a knowledge graph representing the intraoperative scene of a surgery. Node and edge features of the knowledge graph are extracted from suitable data sources in the operating room using machine learning. A spatiotemporal graph neural network architecture subsequently allows for interpretation of relational and temporal patterns within the knowledge graph. We apply our approach to the downstream task of instrument anticipation while presenting a suitable modeling and evaluation strategy for this task.
Results
Our approach achieves an F1 score of 66.86% in terms of instrument anticipation, allowing for a seamless surgical workflow and adding a valuable impact for surgical decision support systems. A resting recall of 63.33% indicates the non-prematurity of the anticipations.
Conclusion
This work shows how multimodal data can be combined with the topological properties of an operating room in a graph-based approach. Our multimodal graph architecture serves as a basis for context-sensitive decision support systems in laparoscopic surgery considering a comprehensive intraoperative operating scene.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Munich, Germany. lars.wagner@tum.de.
- Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Research Group MITI, Munich, Germany.
- Technical University of Munich, TUM School of Medicine and Health, Klinikum rechts der Isar, Department of Surgery, Munich, Germany.
- Technical University of Munich, TUM School of Computation, Information and Technology, Chair of Robotics, Artificial Intelligence and Real-Time Systems, Munich, Germany.
Förderung
- Bayerische Staatsministerium für Wirtschaft, Landesentwicklung und Energie
Zitieren
APA
Wagner, L., Schneider, D. N., Mayer, L., Jell, A., Müller, C., Lenz, A., Knoll, A., & Wilhelm, D. (2024). Towards multimodal graph neural networks for surgical instrument anticipation. International Journal of Computer Assisted Radiology and Surgery, 19(10), 1929-1937. https://doi.org/10.1007/s11548-024-03226-8
Vancouver
Wagner L, Schneider DN, Mayer L, Jell A, Müller C, Lenz A, et al. Towards multimodal graph neural networks for surgical instrument anticipation. International Journal of Computer Assisted Radiology and Surgery. 2024;19(10):1929-1937. doi:10.1007/s11548-024-03226-8
BibTeX
@article{Wagner2024,
author = {Wagner, Lars and Schneider, Dennis N and Mayer, Leon and Jell, Alissa and Müller, Carolin and Lenz, Alexander and Knoll, Alois and Wilhelm, Dirk},
title = {Towards multimodal graph neural networks for surgical instrument anticipation},
journal = {International Journal of Computer Assisted Radiology and Surgery},
year = {2024},
volume = {19},
number = {10},
pages = {1929--1937},
doi = {10.1007/s11548-024-03226-8}
}
EUVIMED