Evaluating the Robustness of a Deep Learning Bone Age Algorithm to Clinical Image Variation Using Computational Stress Testing
Radiology: Artificial Intelligence · 2024
DOI: 10.1148/ryai.230240 ↗Kurz gefasst
Fachartikel in der Zeitschrift Radiology: Artificial Intelligence, 2024. Verfasst von Samantha M Santomartino und 4 weiteren Autor:innen. Beteiligte Einrichtung: From the Drexel University College of Medicine, Philadelphia, Pa (S.M.S.); University of Maryland Medical Intelligent Imaging (UM2ii) Center, Departm…. Hauptthemen laut MEDLINE: Deep Learning, Age Determination by Skeleton. Die Arbeit wurde bisher 19-mal zitiert (Crossref) und nennt 16 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Purpose
To evaluate the robustness of an award-winning bone age deep learning (DL) model to extensive variations in image appearance. Materials and Methods In December 2021, the DL bone age model that won the 2017 RSNA Pediatric Bone Age Challenge was retrospectively evaluated using the RSNA validation set (1425 pediatric hand radiographs; internal test set in this study) and the Digital Hand Atlas (DHA) (1202 pediatric hand radiographs; external test set). Each test image underwent seven types of transformations (rotations, flips, brightness, contrast, inversion, laterality marker, and resolution) to represent a range of image appearances, many of which simulate real-world variations. Computational "stress tests" were performed by comparing the model's predictions on baseline and transformed images. Mean absolute differences (MADs) of predicted bone ages compared with radiologist-determined ground truth on baseline versus transformed images were compared using Wilcoxon signed rank tests. The proportion of clinically significant errors (CSEs) was compared using McNemar tests.
Results
There was no evidence of a difference in MAD of the model on the two baseline test sets (RSNA = 6.8 months, DHA = 6.9 months; P = .05), indicating good model generalization to external data. Except for the RSNA dataset images with an appended radiologic laterality marker (P = .86), there were significant differences in MAD for both the DHA and RSNA datasets among other transformation groups (rotations, flips, brightness, contrast, inversion, and resolution). There were significant differences in proportion of CSEs for 57% of the image transformations (19 of 33) performed on the DHA dataset.
Conclusion
Although an award-winning pediatric bone age DL model generalized well to curated external images, it had inconsistent predictions on images that had undergone simple transformations reflective of several real-world variations in image appearance. Keywords: Pediatrics, Hand, Convolutional Neural Network, Radiography Supplemental material is available for this article. © RSNA, 2024 See also commentary by Faghani and Erickson in this issue.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- From the Drexel University College of Medicine, Philadelphia, Pa (S.M.S.); University of Maryland Medical Intelligent Imaging (UM2ii) Center, Department of Diagnostic Radiology and Nuclear Medicine, University of Maryla…
ORCID-Profile: Samantha M Santomartino, Kristin Putman, Elham Beheshtian, Paul H Yi
Zitieren
APA
Santomartino, S. M., Putman, K., Beheshtian, E., Parekh, V. S., & Yi, P. H. (2024). Evaluating the Robustness of a Deep Learning Bone Age Algorithm to Clinical Image Variation Using Computational Stress Testing. Radiology: Artificial Intelligence. https://doi.org/10.1148/ryai.230240
Vancouver
Santomartino SM, Putman K, Beheshtian E, Parekh VS, Yi PH. Evaluating the Robustness of a Deep Learning Bone Age Algorithm to Clinical Image Variation Using Computational Stress Testing. Radiology: Artificial Intelligence. 2024. doi:10.1148/ryai.230240
BibTeX
@article{Santomartino2024,
author = {Santomartino, Samantha M and Putman, Kristin and Beheshtian, Elham and Parekh, Vishwa S and Yi, Paul H},
title = {Evaluating the Robustness of a Deep Learning Bone Age Algorithm to Clinical Image Variation Using Computational Stress Testing},
journal = {Radiology: Artificial Intelligence},
year = {2024},
doi = {10.1148/ryai.230240}
}
EUVIMED