Surgical instrument-tissue interaction recognition with multi-task-attention video transformer
International Journal of Computer Assisted Radiology and Surgery · 2025
DOI: 10.1007/s11548-025-03546-3 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Computer Assisted Radiology and Surgery, 2025. Verfasst von Lennart Maack und 4 weiteren Autor:innen. Beteiligte Einrichtung: Institute of Medical Technology and Intelligent Systems, Hamburg University of Technology, Hamburg, Germany. lennart.maack@tuhh.de. Hauptthemen laut MEDLINE: Video Recording, Surgical Instruments, Deep Learning, Minimally Invasive Surgical Procedures. Die Arbeit nennt 26 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Horizon Europe programme.
Abstract (Originalsprache)
Purpose
The recognition of surgical instrument-tissue interactions can enhance the surgical workflow analysis, improve automated safety systems and enable skill assessment in minimally invasive surgery. However, current deep learning methods for surgical instrument-tissue interaction recognition often rely on static images or coarse temporal sampling, limiting their ability to capture rapid surgical dynamics. Therefore, this study systematically investigates the impact of incorporating fine-grained temporal context into deep learning models for interaction recognition.
Methods
We conduct extensive experiments with multiple curated video-based datasets to investigate the influence of fine-grained temporal context for the task of instrument-tissue interaction recognition using video transformer with spatio-temporal feature extraction capabilities. Additionally, we propose a multi-task-attention module that utilizes cross-attention and a gating mechanism to improve communication between the subtasks of identifying the surgical instrument, atomic action, and anatomical target.
Results
Our study demonstrates the benefit of utilizing the fine-grained temporal context for recognition of instrument-tissue interactions, with an optimal sampling rate of 6-8 Hz identified for the examined datasets. Furthermore, our proposed MTAM significantly outperforms state-of-the-art multi-task video transformer on the CholecT45-Vid and GraSP-Vid datasets, achieving relative increases of $$4.8 \%$$ 4.8 % and $$5.9 \%$$ 5.9 % in surgical instrument-tissue interaction recognition, respectively.
Conclusions
In this work, we demonstrate the benefits of using a fine-grained temporal context rather than static images or coarse temporal context for the task of surgical instrument-tissue interaction recognition. We also show that leveraging cross-attention with spatio-temporal features from various subtasks leads to improved surgical instrument-tissue interaction recognition performance. The project is available at: https://lennart-maack.github.io/InstrTissRec-MTAM .
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Institute of Medical Technology and Intelligent Systems, Hamburg University of Technology, Hamburg, Germany. lennart.maack@tuhh.de.
- Institute of Medical Technology and Intelligent Systems, Hamburg University of Technology, Hamburg, Germany.
- Martini-Klinik Prostate Cancer Center, University Medical Center Hamburg-Eppendorf, Hamburg, Germany.
- Department of Urology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany.
Förderung
- Horizon Europe programme
Zitieren
APA
Maack, L., Cam, B., Latus, S., Maurer, T., & Schlaefer, A. (2025). Surgical instrument-tissue interaction recognition with multi-task-attention video transformer. International Journal of Computer Assisted Radiology and Surgery. https://doi.org/10.1007/s11548-025-03546-3
Vancouver
Maack L, Cam B, Latus S, Maurer T, Schlaefer A. Surgical instrument-tissue interaction recognition with multi-task-attention video transformer. International Journal of Computer Assisted Radiology and Surgery. 2025. doi:10.1007/s11548-025-03546-3
BibTeX
@article{Maack2025,
author = {Maack, Lennart and Cam, Berk and Latus, Sarah and Maurer, Tobias and Schlaefer, Alexander},
title = {Surgical instrument-tissue interaction recognition with multi-task-attention video transformer},
journal = {International Journal of Computer Assisted Radiology and Surgery},
year = {2025},
doi = {10.1007/s11548-025-03546-3}
}
EUVIMED