Class imbalance on medical image classification: towards better evaluation practices for discrimination and calibration performance
European Radiology · 11.06.2024 · 34 (12) · 7895-7903
DOI: 10.1007/s00330-024-10834-0 ↗Kurz gefasst
Übersichtsarbeit in der Zeitschrift European Radiology, online erschienen am 11.06.2024. Verfasst von Candelaria Mosquera und 4 weiteren Autor:innen. Beteiligte Einrichtung: Hospital Italiano de Buenos Aires, Buenos Aires, Argentina. candelaria.mosquera@hospitalitaliano.org.ar. Hauptthemen laut MEDLINE: Radiography, Thoracic. Die Arbeit wurde bisher 90-mal zitiert (Crossref) und nennt 38 Literaturangaben. Gefördert von: Agencia Nacional de Promoción Científica y Tecnológica.
Abstract (Originalsprache)
PURPOSE: This work aims to assess standard evaluation practices used by the research community for evaluating medical imaging classifiers, with a specific focus on the implications of class imbalance. The analysis is performed on chest X-rays as a case study and encompasses a comprehensive model performance definition, considering both discriminative capabilities and model calibration. MATERIALS AND METHODS: We conduct a concise literature review to examine prevailing scientific practices used when evaluating X-ray classifiers. Then, we perform a systematic experiment on two major chest X-ray datasets to showcase a didactic example of the behavior of several performance metrics under different class ratios and highlight how widely adopted metrics can conceal performance in the minority class. RESULTS: Our literature study confirms that: (1) even when dealing with highly imbalanced datasets, the community tends to use metrics that are dominated by the majority class; and (2) it is still uncommon to include calibration studies for chest X-ray classifiers, albeit its importance in the context of healthcare. Moreover, our systematic experiments confirm that current evaluation practices may not reflect model performance in real clinical scenarios and suggest complementary metrics to better reflect the performance of the system in such scenarios. CONCLUSION: Our analysis underscores the need for enhanced evaluation practices, particularly in the context of class-imbalanced chest X-ray classifiers. We recommend the inclusion of complementary metrics such as the area under the precision-recall curve (AUC-PR), adjusted AUC-PR, and balanced Brier score, to offer a more accurate depiction of system performance in real clinical scenarios, considering metrics that reflect both, discrimination and calibration performance. CLINICAL RELEVANCE STATEMENT: This study underscores the critical need for refined evaluation metrics in medical imaging classifiers, emphasizing that prevalent metrics may mask poor performance in minority classes, potentially impacting clinical diagnoses and healthcare outcomes. KEY POINTS: Common scientific practices in papers dealing with X-ray computer-assisted diagnosis (CAD) systems may be misleading. We highlight limitations in reporting of evaluation metrics for X-ray CAD systems in highly imbalanced scenarios. We propose adopting alternative metrics based on experimental evaluation on large-scale datasets.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Hospital Italiano de Buenos Aires, Buenos Aires, Argentina. candelaria.mosquera@hospitalitaliano.org.ar.
- Universidad Tecnológica Nacional, Buenos Aires, Argentina. candelaria.mosquera@hospitalitaliano.org.ar.
- Instituto de Ciencias de la Computación, UBA-CONICET, Buenos Aires, Argentina.
- Institute for Signals, Systems, and Computational Intelligence, sinc(i) CONICET-UNL, Santa Fe, Argentina.
- Hospital Italiano de Buenos Aires, Buenos Aires, Argentina.
- Institute for Signals, Systems, and Computational Intelligence, sinc(i) CONICET-UNL, Santa Fe, Argentina. eferrante@sinc.unl.edu.ar.
Förderung
- Agencia Nacional de Promoción Científica y Tecnológica
Zitieren
APA
Mosquera, C., Ferrer, L., Milone, D. H., Luna, D., & Ferrante, E. (2024). Class imbalance on medical image classification: towards better evaluation practices for discrimination and calibration performance. European Radiology, 34(12), 7895-7903. https://doi.org/10.1007/s00330-024-10834-0
Vancouver
Mosquera C, Ferrer L, Milone DH, Luna D, Ferrante E. Class imbalance on medical image classification: towards better evaluation practices for discrimination and calibration performance. European Radiology. 2024;34(12):7895-7903. doi:10.1007/s00330-024-10834-0
BibTeX
@article{Mosquera2024,
author = {Mosquera, Candelaria and Ferrer, Luciana and Milone, Diego H and Luna, Daniel and Ferrante, Enzo},
title = {Class imbalance on medical image classification: towards better evaluation practices for discrimination and calibration performance},
journal = {European Radiology},
year = {2024},
volume = {34},
number = {12},
pages = {7895--7903},
doi = {10.1007/s00330-024-10834-0}
}
EUVIMED