EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

EndoViT: pretraining vision transformers on a large collection of endoscopic images

Dominik Batić, Felix Holm, Ege Özsoy, Tobias Czempiel, Nassir Navab

International Journal of Computer Assisted Radiology and Surgery · 2024

DOI: 10.1007/s11548-024-03091-5 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Computer Assisted Radiology and Surgery, 2024. Verfasst von Dominik Batić und 4 weiteren Autor:innen. Beteiligte Einrichtung: Chair for Computer Aided Medical Procedures, Technical University Munich, Munich, Germany. Hauptthemen laut MEDLINE: Endoscopy, Automation, Natural Language Processing, Thoracic Surgery, Video-Assisted. Die Arbeit wurde bisher 42-mal zitiert (Crossref) und nennt 30 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Stryker, Carl Zeiss AG.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Purpose

Automated endoscopy video analysis is essential for assisting surgeons during medical procedures, but it faces challenges due to complex surgical scenes and limited annotated data. Large-scale pretraining has shown great success in natural language processing and computer vision communities in recent years. These approaches reduce the need for annotated data, which is of great interest in the medical domain. In this work, we investigate endoscopy domain-specific self-supervised pretraining on large collections of data.

Methods

To this end, we first collect Endo700k, the largest publicly available corpus of endoscopic images, extracted from nine public Minimally Invasive Surgery (MIS) datasets. Endo700k comprises more than 700,000 images. Next, we introduce EndoViT, an endoscopy-pretrained Vision Transformer (ViT), and evaluate it on a diverse set of surgical downstream tasks.

Results

Our findings indicate that domain-specific pretraining with EndoViT yields notable advantages in complex downstream tasks. In the case of action triplet recognition, our approach outperforms ImageNet pretraining. In semantic segmentation, we surpass the state-of-the-art (SOTA) performance. These results demonstrate the effectiveness of our domain-specific pretraining approach in addressing the challenges of automated endoscopy video analysis.

Conclusion

Our study contributes to the field of medical computer vision by showcasing the benefits of domain-specific large-scale self-supervised pretraining for vision transformers. We release both our code and pretrained models to facilitate further research in this direction: https://github.com/DominikBatic/EndoViT .

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

EndoscopyAutomationNatural Language ProcessingThoracic Surgery, Video-AssistedDiagnostic ImagingHumansMinimally Invasive Surgical ProceduresDeep LearningLarge Language Models

Stichwörter der Autor:innen

Endoscopy video analysisPretrainingVision transformer

Autor:innen und Institutionen

  • Chair for Computer Aided Medical Procedures, Technical University Munich, Munich, Germany.
  • Chair for Computer Aided Medical Procedures, Technical University Munich, Munich, Germany. felix.holm@tum.de.
  • Carl Zeiss AG, Munich, Germany. felix.holm@tum.de.

Förderung

  • Stryker
  • Carl Zeiss AG

Zitieren

APA

Batić, D., Holm, F., Özsoy, E., Czempiel, T., & Navab, N. (2024). EndoViT: pretraining vision transformers on a large collection of endoscopic images. International Journal of Computer Assisted Radiology and Surgery. https://doi.org/10.1007/s11548-024-03091-5

Vancouver

Batić D, Holm F, Özsoy E, Czempiel T, Navab N. EndoViT: pretraining vision transformers on a large collection of endoscopic images. International Journal of Computer Assisted Radiology and Surgery. 2024. doi:10.1007/s11548-024-03091-5

BibTeX
@article{Bati2024,
  author = {Batić, Dominik and Holm, Felix and Özsoy, Ege and Czempiel, Tobias and Navab, Nassir},
  title = {EndoViT: pretraining vision transformers on a large collection of endoscopic images},
  journal = {International Journal of Computer Assisted Radiology and Surgery},
  year = {2024},
  doi = {10.1007/s11548-024-03091-5}
}
Kostenlos starten →