EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Quantifying Epistemic Uncertainty in Predictions for Safer Health AI Performance Under Dataset Shifts

David Fernández-Narro, Pablo Ferri, Juan Miguel García-Gómez, Carlos Sáez

Studies in Health Technology and Informatics · 2025

DOI: 10.3233/shti251493 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Studies in Health Technology and Informatics, 2025. Verfasst von David Fernández-Narro und 3 weiteren Autor:innen. Beteiligte Einrichtung: Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones, Universitat Politècnica de Valéncia, Valencia…. Hauptthemen laut MEDLINE: COVID-19, Decision Support Systems, Clinical, Artificial Intelligence. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Out-of-distribution data , data coming from a different distribution with respect to the training data, entails a critical challenge for the robustness and safety of AI-based clinical decision support systems (CDSSs). This work aims to investigate whether real-time, sample-level quantification of epistemic uncertainty, the model’s uncertainty due to limited knowledge of the true data-generating process, can act as a lightweight safety layer for health AI and CDSSs, targeting model updates and spotlighting human review. To this end, we trained and evaluated a continual learning-based neural network classifier on quarterly batches in a real-world Mexican COVID-19 dataset. For each training window, we estimated the distribution of the prediction epistemic uncertainties using Monte Carlo Dropout. We set a data-driven uncertainty threshold to determine potential out-of-distribution samples at 95% of that distribution. Results across all training–test time pairs show that samples below this threshold exhibit consistently higher macro-F1 and render performance virtually invariant to temporal drift, while the flagged samples captured most prediction errors. Since our method requires no model retraining, sample-level epistemic uncertainty screening offers a practical and efficient first line of defense for deploying health-AI systems in dynamic environments.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

COVID-19Decision Support Systems, ClinicalArtificial IntelligenceUncertaintyHumansNeural Networks, ComputerMexicoSARS-CoV-2

Stichwörter der Autor:innen

Dataset shiftsHealth AIOut-of-distributionTrustworthy AI

Autor:innen und Institutionen

  • Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones, Universitat Politècnica de Valéncia, Valencia, Spain.

Zitieren

APA

Fernández-Narro, D., Ferri, P., García-Gómez, J. M., & Sáez, C. (2025). Quantifying Epistemic Uncertainty in Predictions for Safer Health AI Performance Under Dataset Shifts. Studies in Health Technology and Informatics. https://doi.org/10.3233/shti251493

Vancouver

Fernández-Narro D, Ferri P, García-Gómez JM, Sáez C. Quantifying Epistemic Uncertainty in Predictions for Safer Health AI Performance Under Dataset Shifts. Studies in Health Technology and Informatics. 2025. doi:10.3233/shti251493

BibTeX
@article{FernndezNarro2025,
  author = {Fernández-Narro, David and Ferri, Pablo and García-Gómez, Juan Miguel and Sáez, Carlos},
  title = {Quantifying Epistemic Uncertainty in Predictions for Safer Health AI Performance Under Dataset Shifts},
  journal = {Studies in Health Technology and Informatics},
  year = {2025},
  doi = {10.3233/shti251493}
}
Kostenlos starten →