EndoViT: pretraining vision transformers on a large collection of endoscopic images
International Journal of Computer Assisted Radiology and Surgery · 2024
DOI: 10.1007/s11548-024-03091-5 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Computer Assisted Radiology and Surgery, 2024. Verfasst von Dominik Batić und 4 weiteren Autor:innen. Beteiligte Einrichtung: Chair for Computer Aided Medical Procedures, Technical University Munich, Munich, Germany. Hauptthemen laut MEDLINE: Endoscopy, Automation, Natural Language Processing, Thoracic Surgery, Video-Assisted. Die Arbeit wurde bisher 42-mal zitiert (Crossref) und nennt 30 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Stryker, Carl Zeiss AG.
Abstract (Originalsprache)
Purpose
Automated endoscopy video analysis is essential for assisting surgeons during medical procedures, but it faces challenges due to complex surgical scenes and limited annotated data. Large-scale pretraining has shown great success in natural language processing and computer vision communities in recent years. These approaches reduce the need for annotated data, which is of great interest in the medical domain. In this work, we investigate endoscopy domain-specific self-supervised pretraining on large collections of data.
Methods
To this end, we first collect Endo700k, the largest publicly available corpus of endoscopic images, extracted from nine public Minimally Invasive Surgery (MIS) datasets. Endo700k comprises more than 700,000 images. Next, we introduce EndoViT, an endoscopy-pretrained Vision Transformer (ViT), and evaluate it on a diverse set of surgical downstream tasks.
Results
Our findings indicate that domain-specific pretraining with EndoViT yields notable advantages in complex downstream tasks. In the case of action triplet recognition, our approach outperforms ImageNet pretraining. In semantic segmentation, we surpass the state-of-the-art (SOTA) performance. These results demonstrate the effectiveness of our domain-specific pretraining approach in addressing the challenges of automated endoscopy video analysis.
Conclusion
Our study contributes to the field of medical computer vision by showcasing the benefits of domain-specific large-scale self-supervised pretraining for vision transformers. We release both our code and pretrained models to facilitate further research in this direction: https://github.com/DominikBatic/EndoViT .
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Chair for Computer Aided Medical Procedures, Technical University Munich, Munich, Germany.
- Chair for Computer Aided Medical Procedures, Technical University Munich, Munich, Germany. felix.holm@tum.de.
- Carl Zeiss AG, Munich, Germany. felix.holm@tum.de.
Förderung
- Stryker
- Carl Zeiss AG
Zitieren
APA
Batić, D., Holm, F., Özsoy, E., Czempiel, T., & Navab, N. (2024). EndoViT: pretraining vision transformers on a large collection of endoscopic images. International Journal of Computer Assisted Radiology and Surgery. https://doi.org/10.1007/s11548-024-03091-5
Vancouver
Batić D, Holm F, Özsoy E, Czempiel T, Navab N. EndoViT: pretraining vision transformers on a large collection of endoscopic images. International Journal of Computer Assisted Radiology and Surgery. 2024. doi:10.1007/s11548-024-03091-5
BibTeX
@article{Bati2024,
author = {Batić, Dominik and Holm, Felix and Özsoy, Ege and Czempiel, Tobias and Navab, Nassir},
title = {EndoViT: pretraining vision transformers on a large collection of endoscopic images},
journal = {International Journal of Computer Assisted Radiology and Surgery},
year = {2024},
doi = {10.1007/s11548-024-03091-5}
}
EUVIMED