EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Using Masked Image Modelling Transformer Architecture for Laparoscopic Surgical Tool Classification and Localization

Hisham ElMoaqet iD, Rami Janini iD, Mutaz Ryalat iD, Ghaith Al-Refai iD, Tamer Abdulbaki Alshirbaji iD, Nour Aldeen Jalal iD, Thomas Neumuth iD, Knut Moeller iD, Nassir Navab

Sensors · 10.05.2025 · 25 (10) · 3017

DOI: 10.3390/s25103017 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Sensors, online erschienen am 10.05.2025. Verfasst von Hisham ElMoaqet und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Mechatronics Engineering, German Jordanian University, Amman 11180, Jordan. Hauptthemen laut MEDLINE: Laparoscopy, Image Processing, Computer-Assisted. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 26 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: German Federal Ministry of Research and Education, German Jordanian University Study Group Mobility Program (funded by DAAD), German Federal Ministry of Research and Education (BMBF) under project CoHMed/DigiMedOP.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Artificial intelligence (AI) has shown its potential to advance applications in various medical fields. One such area involves developing integrated AI-based systems to assist in laparoscopic surgery. Surgical tool detection and phase recognition are key components to develop such systems, and therefore, they have been extensively studied in recent years. Despite significant advancements in this field, previous image-based methods still face many challenges that limit their performance due to complex surgical scenes and limited annotated data. This study proposes a novel deep learning approach for classifying and localizing surgical tools in laparoscopic surgeries. The proposed approach uses a self-supervised learning algorithm for surgical tool classification followed by a weakly supervised algorithm for surgical tool localization, eliminating the need for explicit localization annotation. In particular, we leverage the Bidirectional Encoder Representation from Image Transformers (BEiT) model for tool classification and then utilize the heat maps generated from the multi-headed attention layers in the BEiT model for the localizing of these tools. Furthermore, the model incorporates class weights to address the class imbalance issue resulting from different usage frequencies of surgical tools in surgeries. Evaluated on the Cholec80 benchmark dataset, the proposed approach demonstrated high performance in surgical tool classification, surpassing previous works that utilize both spatial and temporal information. Additionally, the proposed weakly supervised learning approach achieved state-of-the-art results for the localization task.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

LaparoscopyImage Processing, Computer-AssistedHumansAlgorithmsArtificial IntelligenceDeep Learning

Stichwörter der Autor:innen

computer-aided surgerieslaparoscopic video analysissurgical tool classificationtool localization

Autor:innen und Institutionen

  • Department of Mechatronics Engineering, German Jordanian University, Amman 11180, Jordan.
  • Department of Electrical Engineering, German Jordanian University, Amman 11180, Jordan.
  • Institute of Technical Medicine (ITeM), Furtwangen University, 78054 Villingen-Schwenningen, Germany.
  • Innovation Center Computer Assisted Surgery (ICCAS), University of Leipzig, 04103 Leipzig, Germany.
  • Department of Microsystems Engineering, University of Freiburg, 79110 Freiburg, Germany.
  • Computer Aided Medical Procedures (CAMP), Technical University of Munich, 85748 Munich, Germany.

ORCID-Profile: Hisham ElMoaqet, Rami Janini, Mutaz Ryalat, Ghaith Al-Refai, Tamer Abdulbaki Alshirbaji, Nour Aldeen Jalal, Thomas Neumuth, Knut Moeller

Förderung

  • German Federal Ministry of Research and Education
  • German Jordanian University Study Group Mobility Program (funded by DAAD)
  • German Federal Ministry of Research and Education (BMBF) under project CoHMed/DigiMedOP
  • PersonaMed

Zitieren

APA

ElMoaqet, H., Janini, R., Ryalat, M., Al-Refai, G., Abdulbaki Alshirbaji, T., Jalal, N. A., Neumuth, T., Moeller, K., & Navab, N. (2025). Using Masked Image Modelling Transformer Architecture for Laparoscopic Surgical Tool Classification and Localization. Sensors, 25(10), 3017. https://doi.org/10.3390/s25103017

Vancouver

ElMoaqet H, Janini R, Ryalat M, Al-Refai G, Abdulbaki Alshirbaji T, Jalal NA, et al. Using Masked Image Modelling Transformer Architecture for Laparoscopic Surgical Tool Classification and Localization. Sensors. 2025;25(10):3017. doi:10.3390/s25103017

BibTeX
@article{ElMoaqet2025,
  author = {ElMoaqet, Hisham and Janini, Rami and Ryalat, Mutaz and Al-Refai, Ghaith and Abdulbaki Alshirbaji, Tamer and Jalal, Nour Aldeen and Neumuth, Thomas and Moeller, Knut and Navab, Nassir},
  title = {Using Masked Image Modelling Transformer Architecture for Laparoscopic Surgical Tool Classification and Localization},
  journal = {Sensors},
  year = {2025},
  volume = {25},
  number = {10},
  pages = {3017},
  doi = {10.3390/s25103017}
}
Kostenlos starten →