EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Surgical instrument-tissue interaction recognition with multi-task-attention video transformer

Lennart Maack, Berk Cam, Sarah Latus, Tobias Maurer, Alexander Schlaefer

International Journal of Computer Assisted Radiology and Surgery · 2025

DOI: 10.1007/s11548-025-03546-3 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Computer Assisted Radiology and Surgery, 2025. Verfasst von Lennart Maack und 4 weiteren Autor:innen. Beteiligte Einrichtung: Institute of Medical Technology and Intelligent Systems, Hamburg University of Technology, Hamburg, Germany. lennart.maack@tuhh.de. Hauptthemen laut MEDLINE: Video Recording, Surgical Instruments, Deep Learning, Minimally Invasive Surgical Procedures. Die Arbeit nennt 26 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Horizon Europe programme.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Purpose

The recognition of surgical instrument-tissue interactions can enhance the surgical workflow analysis, improve automated safety systems and enable skill assessment in minimally invasive surgery. However, current deep learning methods for surgical instrument-tissue interaction recognition often rely on static images or coarse temporal sampling, limiting their ability to capture rapid surgical dynamics. Therefore, this study systematically investigates the impact of incorporating fine-grained temporal context into deep learning models for interaction recognition.

Methods

We conduct extensive experiments with multiple curated video-based datasets to investigate the influence of fine-grained temporal context for the task of instrument-tissue interaction recognition using video transformer with spatio-temporal feature extraction capabilities. Additionally, we propose a multi-task-attention module that utilizes cross-attention and a gating mechanism to improve communication between the subtasks of identifying the surgical instrument, atomic action, and anatomical target.

Results

Our study demonstrates the benefit of utilizing the fine-grained temporal context for recognition of instrument-tissue interactions, with an optimal sampling rate of 6-8 Hz identified for the examined datasets. Furthermore, our proposed MTAM significantly outperforms state-of-the-art multi-task video transformer on the CholecT45-Vid and GraSP-Vid datasets, achieving relative increases of $$4.8 \%$$ 4.8 % and $$5.9 \%$$ 5.9 % in surgical instrument-tissue interaction recognition, respectively.

Conclusions

In this work, we demonstrate the benefits of using a fine-grained temporal context rather than static images or coarse temporal context for the task of surgical instrument-tissue interaction recognition. We also show that leveraging cross-attention with spatio-temporal features from various subtasks leads to improved surgical instrument-tissue interaction recognition performance. The project is available at: https://lennart-maack.github.io/InstrTissRec-MTAM .

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Video RecordingSurgical InstrumentsDeep LearningMinimally Invasive Surgical ProceduresHumans

Stichwörter der Autor:innen

Deep learningSurgical activity recognitionSurgical triplet recognitionVideo transformer

Autor:innen und Institutionen

  • Institute of Medical Technology and Intelligent Systems, Hamburg University of Technology, Hamburg, Germany. lennart.maack@tuhh.de.
  • Institute of Medical Technology and Intelligent Systems, Hamburg University of Technology, Hamburg, Germany.
  • Martini-Klinik Prostate Cancer Center, University Medical Center Hamburg-Eppendorf, Hamburg, Germany.
  • Department of Urology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany.

Förderung

  • Horizon Europe programme

Zitieren

APA

Maack, L., Cam, B., Latus, S., Maurer, T., & Schlaefer, A. (2025). Surgical instrument-tissue interaction recognition with multi-task-attention video transformer. International Journal of Computer Assisted Radiology and Surgery. https://doi.org/10.1007/s11548-025-03546-3

Vancouver

Maack L, Cam B, Latus S, Maurer T, Schlaefer A. Surgical instrument-tissue interaction recognition with multi-task-attention video transformer. International Journal of Computer Assisted Radiology and Surgery. 2025. doi:10.1007/s11548-025-03546-3

BibTeX
@article{Maack2025,
  author = {Maack, Lennart and Cam, Berk and Latus, Sarah and Maurer, Tobias and Schlaefer, Alexander},
  title = {Surgical instrument-tissue interaction recognition with multi-task-attention video transformer},
  journal = {International Journal of Computer Assisted Radiology and Surgery},
  year = {2025},
  doi = {10.1007/s11548-025-03546-3}
}
Kostenlos starten →