Using Masked Image Modelling Transformer Architecture for Laparoscopic Surgical Tool Classification and Localization
Sensors · 10.05.2025 · 25 (10) · 3017
DOI: 10.3390/s25103017 ↗Kurz gefasst
Fachartikel in der Zeitschrift Sensors, online erschienen am 10.05.2025. Verfasst von Hisham ElMoaqet und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Mechatronics Engineering, German Jordanian University, Amman 11180, Jordan. Hauptthemen laut MEDLINE: Laparoscopy, Image Processing, Computer-Assisted. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 26 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: German Federal Ministry of Research and Education, German Jordanian University Study Group Mobility Program (funded by DAAD), German Federal Ministry of Research and Education (BMBF) under project CoHMed/DigiMedOP.
Abstract (Originalsprache)
Artificial intelligence (AI) has shown its potential to advance applications in various medical fields. One such area involves developing integrated AI-based systems to assist in laparoscopic surgery. Surgical tool detection and phase recognition are key components to develop such systems, and therefore, they have been extensively studied in recent years. Despite significant advancements in this field, previous image-based methods still face many challenges that limit their performance due to complex surgical scenes and limited annotated data. This study proposes a novel deep learning approach for classifying and localizing surgical tools in laparoscopic surgeries. The proposed approach uses a self-supervised learning algorithm for surgical tool classification followed by a weakly supervised algorithm for surgical tool localization, eliminating the need for explicit localization annotation. In particular, we leverage the Bidirectional Encoder Representation from Image Transformers (BEiT) model for tool classification and then utilize the heat maps generated from the multi-headed attention layers in the BEiT model for the localizing of these tools. Furthermore, the model incorporates class weights to address the class imbalance issue resulting from different usage frequencies of surgical tools in surgeries. Evaluated on the Cholec80 benchmark dataset, the proposed approach demonstrated high performance in surgical tool classification, surpassing previous works that utilize both spatial and temporal information. Additionally, the proposed weakly supervised learning approach achieved state-of-the-art results for the localization task.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Mechatronics Engineering, German Jordanian University, Amman 11180, Jordan.
- Department of Electrical Engineering, German Jordanian University, Amman 11180, Jordan.
- Institute of Technical Medicine (ITeM), Furtwangen University, 78054 Villingen-Schwenningen, Germany.
- Innovation Center Computer Assisted Surgery (ICCAS), University of Leipzig, 04103 Leipzig, Germany.
- Department of Microsystems Engineering, University of Freiburg, 79110 Freiburg, Germany.
- Computer Aided Medical Procedures (CAMP), Technical University of Munich, 85748 Munich, Germany.
ORCID-Profile: Hisham ElMoaqet, Rami Janini, Mutaz Ryalat, Ghaith Al-Refai, Tamer Abdulbaki Alshirbaji, Nour Aldeen Jalal, Thomas Neumuth, Knut Moeller
Förderung
- German Federal Ministry of Research and Education
- German Jordanian University Study Group Mobility Program (funded by DAAD)
- German Federal Ministry of Research and Education (BMBF) under project CoHMed/DigiMedOP
- PersonaMed
Zitieren
APA
ElMoaqet, H., Janini, R., Ryalat, M., Al-Refai, G., Abdulbaki Alshirbaji, T., Jalal, N. A., Neumuth, T., Moeller, K., & Navab, N. (2025). Using Masked Image Modelling Transformer Architecture for Laparoscopic Surgical Tool Classification and Localization. Sensors, 25(10), 3017. https://doi.org/10.3390/s25103017
Vancouver
ElMoaqet H, Janini R, Ryalat M, Al-Refai G, Abdulbaki Alshirbaji T, Jalal NA, et al. Using Masked Image Modelling Transformer Architecture for Laparoscopic Surgical Tool Classification and Localization. Sensors. 2025;25(10):3017. doi:10.3390/s25103017
BibTeX
@article{ElMoaqet2025,
author = {ElMoaqet, Hisham and Janini, Rami and Ryalat, Mutaz and Al-Refai, Ghaith and Abdulbaki Alshirbaji, Tamer and Jalal, Nour Aldeen and Neumuth, Thomas and Moeller, Knut and Navab, Nassir},
title = {Using Masked Image Modelling Transformer Architecture for Laparoscopic Surgical Tool Classification and Localization},
journal = {Sensors},
year = {2025},
volume = {25},
number = {10},
pages = {3017},
doi = {10.3390/s25103017}
}
EUVIMED