Evaluation of Machine Learning Model Performance in Diabetic Foot Ulcer: Retrospective Cohort Study
Bewertung der Leistung eines Machine-Learning-Modells bei diabetischem Fußulkus: retrospektive Kohortenstudie
JMIR Medical Informatics · 10.10.2025 · 13 · e71994
DOI: 10.2196/71994 ↗Auf einen Blick
In 700 Patient:innen mit diabetischem Fußulkus schnitt ein SVM-Modell bei der Vorhersage der vollständigen Wundheilung am besten ab. Auch RF und XGBoost zeigten gute Leistung und Kalibrierung.
Fragestellung
Wie gut sagen verschiedene Machine-Learning-Modelle die vollständige Wundheilung bei diabetischem Fußulkus vorher?
Methode
Retrospektive Kohortenstudie mit Vergleich von 7 ML-Modellen auf einem DFU-Datensatz; Training, Kreuzvalidierung und Optimierung per Random Sampling; Bewertung mit Kalibrierungskurven, Brier Score und DCA.
Ergebnisse
Explorativer Datensatz mit 700 Patient:innen und 199 Variablen. SVM: Accuracy 0,853 (95 %-KI 0,810–0,896), AUC 0,922 (95 %-KI 0,889–0,955). RF: Accuracy 0,838; AUC 0,917. XGBoost: Accuracy 0,815; AUC 0,889.
Schlussfolgerung der Autor:innen
Nach Ansicht der Autor:innen sind fehlende Werte, Merkmalsauswahl und Klassenungleichgewicht zentrale Schritte bei ML-Anwendungen. In diesem DFU-Datensatz erzielte SVM die beste Leistung, gefolgt von RF und XGBoost.
Kernaussagen
- SVM erreichte die höchste Vorhersageleistung für vollständige Wundheilung.
- RF und XGBoost lagen hinter SVM, zeigten aber ebenfalls gute Werte.
- Kalibrierung und klinischer Nutzen wurden mit Brier Score und DCA geprüft.
Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.
Kurz gefasst
Kohortenstudie in der Zeitschrift JMIR Medical Informatics, online erschienen am 10.10.2025. Verfasst von Veerle Y van Velze und 14 weiteren Autor:innen. Beteiligte Einrichtung: Department of Surgery, Haaglanden Medical Center, The Hague, The Netherlands. Hauptthemen laut MEDLINE: Diabetic Foot, Machine Learning. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 51 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Themen: Diabetisches Fußsyndrom, Künstliche Intelligenz in der Pflege
Abstract (Originalsprache)
Background
Machine learning (ML) has shown great potential in recognizing complex disease patterns and supporting clinical decision-making. Diabetic foot ulcers (DFUs) represent a significant multifactorial medical problem with high incidence and severe outcomes, providing an ideal example for a comprehensive framework that encompasses all essential steps for implementing ML in a clinically relevant fashion.
Objective
This paper aims to provide a framework for the proper use of ML algorithms to predict clinical outcomes of multifactorial diseases and their treatments.
Methods
The comparison of ML models was performed on a DFU dataset. The selection of patient characteristics associated with wound healing was based on outcomes of statistical tests, that is, ANOVA and chi-square test, and validated on expert recommendations. Imputation and balancing of patient records were performed with MIDAS (Multiple Imputation with Denoising Autoencoders) Touch and adaptive synthetic sampling, respectively. Logistic regression, support vector machine (SVM), k-nearest neighbors, random forest (RF), extreme gradient boosting (XGBoost), Bayesian additive regression trees, and artificial neural network were trained, cross-validated, and optimized using random sampling on the patient dataset. To evaluate model calibration and clinical utility, calibration curves, Brier scores, and decision curve analysis (DCA) were performed.
Results
The exploratory dataset consisted of 700 patient records with 199 variables. After dataset cleaning, the variables used for model training included age, smoking status, toe systolic pressure, blood pressure, oxygen saturation, hemoglobin, hemoglobin A1c, estimated glomerular filtration rate, wound location, diabetes type, Texas wound classification, neuropathy, and wound area measurement. The SVM obtained a stable accuracy of 0.853 (95% CI 0.810-0.896) with an area under the receiver operating characteristic curve of 0.922 (95% CI 0.889-0.955). The RF and XGBoost acquired an accuracy of 0.838 (95% CI 0.793-0.883) and 0.815 (95% CI 0.768-0.862), respectively, with areas under the receiver operating characteristic curve of 0.917 (95% CI 0.883-0.951) for RF and 0.889 (95% CI 0.849-0.929) for XGBoost. SVM, RF, and XGBoost were well-calibrated, with average Brier scores around 0.127 (SD 0.013). DCA showed that the SVM provided the highest net clinical benefit across relevant risk thresholds.
Conclusions
Handling missing values, feature selection, and addressing class imbalance are critical components of the key steps in developing ML applications for clinical research. Seven models were selected for comparing their predictive power regarding complete wound healing, and each model representing a different branch in ML. In this initial DFU dataset used as an example, the SVM achieved the best performance in predicting clinical outcomes, followed by RF and XGBoost. The model’s calibration and clinical utility were determined through calibration curves, Brier scores, and DCA, demonstrating its potential relevance in clinical decision-making.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Surgery, Haaglanden Medical Center, The Hague, The Netherlands.
- Centre for Human Drug Research, Leiden, The Netherlands.
- The Hague University of Applied Sciences, The Hague, Zoetermeer, The Netherlands.
- University Vascular Centre West, Leiden, The Hague, Delft, The Netherlands.
- Department of Vascular Surgery, Leiden University Medical Center, Leiden, The Netherlands.
- Department of Dermatology, Leiden University Medical Center, Leiden, The Netherlands.
ORCID-Profile: Veerle Y van Velze, Hendrico L Burger, Tim J van der Steenhoven, Hani Al-Ers, Lauren N Goncalves, Daniël Eefting, Willem-Jan J de Jong, Harm J Smeets, Janna C Specken Welleweerd, Joost R van der Vorst, Sandy Uchtmann, Robert Rissmann, Jaap F Hamming, Lampros Stergioulas, Koen Ea van der Bogt
Zitieren
APA
van Velze, V. Y., Burger, H. L., van der Steenhoven, T. J., Al-Ers, H., Goncalves, L. N., Eefting, D., de Jong, W. J., Smeets, H. J., Welleweerd, J. C. S., van der Vorst, J. R., Uchtmann, S., Rissmann, R., Hamming, J. F., Stergioulas, L., & van der Bogt, K. E. (2025). Evaluation of Machine Learning Model Performance in Diabetic Foot Ulcer: Retrospective Cohort Study. JMIR Medical Informatics, 13, e71994. https://doi.org/10.2196/71994
Vancouver
van Velze VY, Burger HL, van der Steenhoven TJ, Al-Ers H, Goncalves LN, Eefting D, et al. Evaluation of Machine Learning Model Performance in Diabetic Foot Ulcer: Retrospective Cohort Study. JMIR Medical Informatics. 2025;13:e71994. doi:10.2196/71994
BibTeX
@article{vanVelze2025,
author = {van Velze, Veerle Y and Burger, Hendrico L and van der Steenhoven, Tim J and Al-Ers, Hani and Goncalves, Lauren N and Eefting, Daniël and de Jong, Willem-Jan J and Smeets, Harm J and Welleweerd, Janna C Specken and van der Vorst, Joost R and Uchtmann, Sandy and Rissmann, Robert and Hamming, Jaap F and Stergioulas, Lampros and van der Bogt, Koen Ea},
title = {Evaluation of Machine Learning Model Performance in Diabetic Foot Ulcer: Retrospective Cohort Study},
journal = {JMIR Medical Informatics},
year = {2025},
volume = {13},
pages = {e71994},
doi = {10.2196/71994}
}
EUVIMED