Sources of performance variability in deep learning-based polyp detection
International Journal of Computer Assisted Radiology and Surgery · 2023
DOI: 10.1007/s11548-023-02936-9 ↗Kurz gefasst
Fachartikel in der Zeitschrift International Journal of Computer Assisted Radiology and Surgery, 2023. Verfasst von T N Tran und 10 weiteren Autor:innen. Beteiligte Einrichtung: Division of Intelligent Medical Systems, DKFZ, Heidelberg, Germany. t.tran@dkfz-heidelberg.de. Hauptthemen laut MEDLINE: Deep Learning, Colonic Neoplasms. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 23 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Deutsches Krebsforschungszentrum (DKFZ).
Abstract (Originalsprache)
Purpose
Validation metrics are a key prerequisite for the reliable tracking of scientific progress and for deciding on the potential clinical translation of methods. While recent initiatives aim to develop comprehensive theoretical frameworks for understanding metric-related pitfalls in image analysis problems, there is a lack of experimental evidence on the concrete effects of common and rare pitfalls on specific applications. We address this gap in the literature in the context of colon cancer screening.
Methods
Our contribution is twofold. Firstly, we present the winning solution of the Endoscopy Computer Vision Challenge on colon cancer detection, conducted in conjunction with the IEEE International Symposium on Biomedical Imaging 2022. Secondly, we demonstrate the sensitivity of commonly used metrics to a range of hyperparameters as well as the consequences of poor metric choices.
Results
Based on comprehensive validation studies performed with patient data from six clinical centers, we found all commonly applied object detection metrics to be subject to high inter-center variability. Furthermore, our results clearly demonstrate that the adaptation of standard hyperparameters used in the computer vision community does not generally lead to the clinically most plausible results. Finally, we present localization criteria that correspond well to clinical relevance.
Conclusion
We conclude from our study that (1) performance results in polyp detection are highly sensitive to various design choices, (2) common metric configurations do not reflect the clinical need and rely on suboptimal hyperparameters and (3) comparison of performance across datasets can be largely misleading. Our work could be a first step towards reconsidering common validation strategies in deep learning-based colonoscopy and beyond.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Division of Intelligent Medical Systems, DKFZ, Heidelberg, Germany. t.tran@dkfz-heidelberg.de.
- Division of Intelligent Medical Systems, DKFZ, Heidelberg, Germany.
- Faculty of Mathematics and Computer Science, University of Heidelberg, Heidelberg, Germany.
- Interdisciplinary Endoscopy Center (IEZ), University Hospital Heidelberg, Heidelberg, Germany.
- Department of Gastroenterology, Hepatology and Endocrinology, Robert-Bosch Hospital (RBK), Stuttgart, Germany.
- Clinic for General Internal Medicine, Gastroenterology, Hepatology and Infectiology, Pneumology, Klinikum Stuttgart, Stuttgart, Germany.
Förderung
- Deutsches Krebsforschungszentrum (DKFZ)
Zitieren
APA
Tran, T. N., Adler, T. J., Yamlahi, A., Christodoulou, E., Godau, P., Reinke, A., Tizabi, M. D., Sauer, P., Persicke, T., Albert, J. G., & Maier-Hein, L. (2023). Sources of performance variability in deep learning-based polyp detection. International Journal of Computer Assisted Radiology and Surgery. https://doi.org/10.1007/s11548-023-02936-9
Vancouver
Tran TN, Adler TJ, Yamlahi A, Christodoulou E, Godau P, Reinke A, et al. Sources of performance variability in deep learning-based polyp detection. International Journal of Computer Assisted Radiology and Surgery. 2023. doi:10.1007/s11548-023-02936-9
BibTeX
@article{Tran2023,
author = {Tran, T N and Adler, T J and Yamlahi, A and Christodoulou, E and Godau, P and Reinke, A and Tizabi, M D and Sauer, P and Persicke, T and Albert, J G and Maier-Hein, L},
title = {Sources of performance variability in deep learning-based polyp detection},
journal = {International Journal of Computer Assisted Radiology and Surgery},
year = {2023},
doi = {10.1007/s11548-023-02936-9}
}
EUVIMED