EUVIMEDObjevte EUVIMED
KohortenstudieOpen access

Evaluation of Machine Learning Model Performance in Diabetic Foot Ulcer: Retrospective Cohort Study

Bewertung der Leistung eines Machine-Learning-Modells bei diabetischem Fußulkus: retrospektive Kohortenstudie

Veerle Y van Velze iD, Hendrico L Burger iD, Tim J van der Steenhoven iD, Hani Al-Ers iD, Lauren N Goncalves iD, Daniël Eefting iD, Willem-Jan J de Jong iD, Harm J Smeets iD, Janna C Specken Welleweerd iD, Joost R van der Vorst iD, Sandy Uchtmann iD, Robert Rissmann iD and 3 more

JMIR Medical Informatics · 2025-10-10 · 13 · e71994

DOI: 10.2196/71994 ↗

Auf einen Blick

In 700 Patient:innen mit diabetischem Fußulkus schnitt ein SVM-Modell bei der Vorhersage der vollständigen Wundheilung am besten ab. Auch RF und XGBoost zeigten gute Leistung und Kalibrierung.

Fragestellung

Wie gut sagen verschiedene Machine-Learning-Modelle die vollständige Wundheilung bei diabetischem Fußulkus vorher?

Methode

Retrospektive Kohortenstudie mit Vergleich von 7 ML-Modellen auf einem DFU-Datensatz; Training, Kreuzvalidierung und Optimierung per Random Sampling; Bewertung mit Kalibrierungskurven, Brier Score und DCA.

Ergebnisse

Explorativer Datensatz mit 700 Patient:innen und 199 Variablen. SVM: Accuracy 0,853 (95 %-KI 0,810–0,896), AUC 0,922 (95 %-KI 0,889–0,955). RF: Accuracy 0,838; AUC 0,917. XGBoost: Accuracy 0,815; AUC 0,889.

Schlussfolgerung der Autor:innen

Nach Ansicht der Autor:innen sind fehlende Werte, Merkmalsauswahl und Klassenungleichgewicht zentrale Schritte bei ML-Anwendungen. In diesem DFU-Datensatz erzielte SVM die beste Leistung, gefolgt von RF und XGBoost.

Kernaussagen

  • SVM erreichte die höchste Vorhersageleistung für vollständige Wundheilung.
  • RF und XGBoost lagen hinter SVM, zeigten aber ebenfalls gute Werte.
  • Kalibrierung und klinischer Nutzen wurden mit Brier Score und DCA geprüft.

Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.

At a glance

Kohortenstudie in JMIR Medical Informatics, published online on 2025-10-10. Written by Veerle Y van Velze and 14 co-authors. Institution: Department of Surgery, Haaglanden Medical Center, The Hague, The Netherlands. Major MEDLINE topics: Diabetic Foot, Machine Learning. Cited 4 times so far (Crossref); 51 references. The full text is freely available in PubMed Central.

Auf einen BlickAbstraktSubjectsBibliografické údajeCiteRelated studies

Abstract (original language)

Background

Machine learning (ML) has shown great potential in recognizing complex disease patterns and supporting clinical decision-making. Diabetic foot ulcers (DFUs) represent a significant multifactorial medical problem with high incidence and severe outcomes, providing an ideal example for a comprehensive framework that encompasses all essential steps for implementing ML in a clinically relevant fashion.

Objective

This paper aims to provide a framework for the proper use of ML algorithms to predict clinical outcomes of multifactorial diseases and their treatments.

Methods

The comparison of ML models was performed on a DFU dataset. The selection of patient characteristics associated with wound healing was based on outcomes of statistical tests, that is, ANOVA and chi-square test, and validated on expert recommendations. Imputation and balancing of patient records were performed with MIDAS (Multiple Imputation with Denoising Autoencoders) Touch and adaptive synthetic sampling, respectively. Logistic regression, support vector machine (SVM), k-nearest neighbors, random forest (RF), extreme gradient boosting (XGBoost), Bayesian additive regression trees, and artificial neural network were trained, cross-validated, and optimized using random sampling on the patient dataset. To evaluate model calibration and clinical utility, calibration curves, Brier scores, and decision curve analysis (DCA) were performed.

Results

The exploratory dataset consisted of 700 patient records with 199 variables. After dataset cleaning, the variables used for model training included age, smoking status, toe systolic pressure, blood pressure, oxygen saturation, hemoglobin, hemoglobin A1c, estimated glomerular filtration rate, wound location, diabetes type, Texas wound classification, neuropathy, and wound area measurement. The SVM obtained a stable accuracy of 0.853 (95% CI 0.810-0.896) with an area under the receiver operating characteristic curve of 0.922 (95% CI 0.889-0.955). The RF and XGBoost acquired an accuracy of 0.838 (95% CI 0.793-0.883) and 0.815 (95% CI 0.768-0.862), respectively, with areas under the receiver operating characteristic curve of 0.917 (95% CI 0.883-0.951) for RF and 0.889 (95% CI 0.849-0.929) for XGBoost. SVM, RF, and XGBoost were well-calibrated, with average Brier scores around 0.127 (SD 0.013). DCA showed that the SVM provided the highest net clinical benefit across relevant risk thresholds.

Conclusions

Handling missing values, feature selection, and addressing class imbalance are critical components of the key steps in developing ML applications for clinical research. Seven models were selected for comparing their predictive power regarding complete wound healing, and each model representing a different branch in ML. In this initial DFU dataset used as an example, the SVM achieved the best performance in predicting clinical outcomes, followed by RF and XGBoost. The model’s calibration and clinical utility were determined through calibration curves, Brier scores, and DCA, demonstrating its potential relevance in clinical decision-making.

Subjects

MeSH terms (MEDLINE)

Diabetic FootMachine LearningHumansRetrospective StudiesMaleFemaleMiddle AgedAgedAlgorithmsBayes TheoremWound Healing

Author keywords

Bayesian additive regression treesartificial neural networkcomplete wound healingdiabetic foot ulcerextreme gradient boostingk-nearest neighborlogistic regressionmachine learningrandom forestsupport vector machine

Authors and institutions

  • Department of Surgery, Haaglanden Medical Center, The Hague, The Netherlands.
  • Centre for Human Drug Research, Leiden, The Netherlands.
  • The Hague University of Applied Sciences, The Hague, Zoetermeer, The Netherlands.
  • University Vascular Centre West, Leiden, The Hague, Delft, The Netherlands.
  • Department of Vascular Surgery, Leiden University Medical Center, Leiden, The Netherlands.
  • Department of Dermatology, Leiden University Medical Center, Leiden, The Netherlands.

ORCID profiles: Veerle Y van Velze, Hendrico L Burger, Tim J van der Steenhoven, Hani Al-Ers, Lauren N Goncalves, Daniël Eefting, Willem-Jan J de Jong, Harm J Smeets, Janna C Specken Welleweerd, Joost R van der Vorst, Sandy Uchtmann, Robert Rissmann, Jaap F Hamming, Lampros Stergioulas, Koen Ea van der Bogt

Cite

APA

van Velze, V. Y., Burger, H. L., van der Steenhoven, T. J., Al-Ers, H., Goncalves, L. N., Eefting, D., de Jong, W. J., Smeets, H. J., Welleweerd, J. C. S., van der Vorst, J. R., Uchtmann, S., Rissmann, R., Hamming, J. F., Stergioulas, L., & van der Bogt, K. E. (2025). Evaluation of Machine Learning Model Performance in Diabetic Foot Ulcer: Retrospective Cohort Study. JMIR Medical Informatics, 13, e71994. https://doi.org/10.2196/71994

Vancouver

van Velze VY, Burger HL, van der Steenhoven TJ, Al-Ers H, Goncalves LN, Eefting D, et al. Evaluation of Machine Learning Model Performance in Diabetic Foot Ulcer: Retrospective Cohort Study. JMIR Medical Informatics. 2025;13:e71994. doi:10.2196/71994

BibTeX
@article{vanVelze2025,
  author = {van Velze, Veerle Y and Burger, Hendrico L and van der Steenhoven, Tim J and Al-Ers, Hani and Goncalves, Lauren N and Eefting, Daniël and de Jong, Willem-Jan J and Smeets, Harm J and Welleweerd, Janna C Specken and van der Vorst, Joost R and Uchtmann, Sandy and Rissmann, Robert and Hamming, Jaap F and Stergioulas, Lampros and van der Bogt, Koen Ea},
  title = {Evaluation of Machine Learning Model Performance in Diabetic Foot Ulcer: Retrospective Cohort Study},
  journal = {JMIR Medical Informatics},
  year = {2025},
  volume = {13},
  pages = {e71994},
  doi = {10.2196/71994}
}
Start for free →