Comparison of evaluation metrics of deep learning for imbalanced imaging data in osteoarthritis studies
Osteoarthritis and Cartilage · 2023
DOI: 10.1016/j.joca.2023.05.006 ↗Kurz gefasst
Fachartikel in der Zeitschrift Osteoarthritis and Cartilage, 2023. Verfasst von Shen Liu und 13 weiteren Autor:innen. Beteiligte Einrichtung: Department of Epidemiology and Biostatistics, University of Arizona, 1295 N. Martin Ave., Tucson, AZ 85724, USA. Electronic address: shenliu@arizona.…. Hauptthemen laut MEDLINE: Deep Learning, Osteoarthritis, Knee, Cartilage Diseases. Die Arbeit wurde bisher 38-mal zitiert (Crossref) und nennt 15 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: NIA NIH HHS, NIAMS NIH HHS, NHLBI NIH HHS.
Abstract (Originalsprache)
PURPOSE: To compare the evaluation metrics for deep learning methods that were developed using imbalanced imaging data in osteoarthritis studies. MATERIALS AND METHODS: This retrospective study utilized 2996 sagittal intermediate-weighted fat-suppressed knee MRIs with MRI Osteoarthritis Knee Score readings from 2467 participants in the Osteoarthritis Initiative study. We obtained probabilities of the presence of bone marrow lesions (BMLs) from MRIs in the testing dataset at the sub-region (15 sub-regions), compartment, and whole-knee levels based on the trained deep learning models. We compared different evaluation metrics (e.g., receiver operating characteristic (ROC) and precision-recall (PR) curves) in the testing dataset with various class ratios (presence of BMLs vs. absence of BMLs) at these three data levels to assess the model's performance. RESULTS: In a subregion with an extremely high imbalance ratio, the model achieved a ROC-AUC of 0.84, a PR-AUC of 0.10, a sensitivity of 0, and a specificity of 1. CONCLUSION: The commonly used ROC curve is not sufficiently informative, especially in the case of imbalanced data. We provide the following practical suggestions based on our data analysis: 1) ROC-AUC is recommended for balanced data, 2) PR-AUC should be used for moderately imbalanced data (i.e., when the proportion of the minor class is above 5% and less than 50%), and 3) for severely imbalanced data (i.e., when the proportion of the minor class is below 5%), it is not practical to apply a deep learning model, even with the application of techniques addressing imbalanced data issues.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Epidemiology and Biostatistics, University of Arizona, 1295 N. Martin Ave., Tucson, AZ 85724, USA. Electronic address: shenliu@arizona.edu.
- Department of Radiology, University of Erlangen - Nuremberg, Erlangen, Germany; Department of Radiology, Boston University School of Medicine, MA, USA. Electronic address: frank.roemer@uk-erlangen.de.
- Department of Management Information Systems, University of Arizona, AZ, USA. Electronic address: yongge@arizona.edu.
- Department of Epidemiology and Biostatistics, University of Arizona, 1295 N. Martin Ave., Tucson, AZ 85724, USA. Electronic address: edwardjbedrick@arizona.edu.
- University of Arizona Arthritis Center, University of Arizona College of Medicine, Tucson, AZ, USA. Electronic address: lizongming@ortho.arizona.edu.
- Department of Radiology, Boston University School of Medicine, MA, USA. Electronic address: guermazi@bu.edu.
Förderung
- NIA NIH HHS
- NIAMS NIH HHS
- NHLBI NIH HHS
Zitieren
APA
Liu, S., Roemer, F., Ge, Y., Bedrick, E. J., Li, Z. M., Guermazi, A., Sharma, L., Eaton, C., Hochberg, M. C., Hunter, D. J., Nevitt, M. C., Wirth, W., Kent Kwoh, C., & Sun, X. (2023). Comparison of evaluation metrics of deep learning for imbalanced imaging data in osteoarthritis studies. Osteoarthritis and Cartilage. https://doi.org/10.1016/j.joca.2023.05.006
Vancouver
Liu S, Roemer F, Ge Y, Bedrick EJ, Li ZM, Guermazi A, et al. Comparison of evaluation metrics of deep learning for imbalanced imaging data in osteoarthritis studies. Osteoarthritis and Cartilage. 2023. doi:10.1016/j.joca.2023.05.006
BibTeX
@article{Liu2023,
author = {Liu, Shen and Roemer, Frank and Ge, Yong and Bedrick, Edward J and Li, Zong-Ming and Guermazi, Ali and Sharma, Leena and Eaton, Charles and Hochberg, Marc C and Hunter, David J and Nevitt, Michael C and Wirth, Wolfgang and Kent Kwoh, C and Sun, Xiaoxiao},
title = {Comparison of evaluation metrics of deep learning for imbalanced imaging data in osteoarthritis studies},
journal = {Osteoarthritis and Cartilage},
year = {2023},
doi = {10.1016/j.joca.2023.05.006}
}
EUVIMED