Quantifying Epistemic Uncertainty in Predictions for Safer Health AI Performance Under Dataset Shifts
Studies in Health Technology and Informatics · 2025
DOI: 10.3233/shti251493 ↗Kurz gefasst
Fachartikel in der Zeitschrift Studies in Health Technology and Informatics, 2025. Verfasst von David Fernández-Narro und 3 weiteren Autor:innen. Beteiligte Einrichtung: Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones, Universitat Politècnica de Valéncia, Valencia…. Hauptthemen laut MEDLINE: COVID-19, Decision Support Systems, Clinical, Artificial Intelligence. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
Out-of-distribution data , data coming from a different distribution with respect to the training data, entails a critical challenge for the robustness and safety of AI-based clinical decision support systems (CDSSs). This work aims to investigate whether real-time, sample-level quantification of epistemic uncertainty, the model’s uncertainty due to limited knowledge of the true data-generating process, can act as a lightweight safety layer for health AI and CDSSs, targeting model updates and spotlighting human review. To this end, we trained and evaluated a continual learning-based neural network classifier on quarterly batches in a real-world Mexican COVID-19 dataset. For each training window, we estimated the distribution of the prediction epistemic uncertainties using Monte Carlo Dropout. We set a data-driven uncertainty threshold to determine potential out-of-distribution samples at 95% of that distribution. Results across all training–test time pairs show that samples below this threshold exhibit consistently higher macro-F1 and render performance virtually invariant to temporal drift, while the flagged samples captured most prediction errors. Since our method requires no model retraining, sample-level epistemic uncertainty screening offers a practical and efficient first line of defense for deploying health-AI systems in dynamic environments.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Biomedical Data Science Lab, Instituto Universitario de Tecnologías de la Información y Comunicaciones, Universitat Politècnica de Valéncia, Valencia, Spain.
Zitieren
APA
Fernández-Narro, D., Ferri, P., García-Gómez, J. M., & Sáez, C. (2025). Quantifying Epistemic Uncertainty in Predictions for Safer Health AI Performance Under Dataset Shifts. Studies in Health Technology and Informatics. https://doi.org/10.3233/shti251493
Vancouver
Fernández-Narro D, Ferri P, García-Gómez JM, Sáez C. Quantifying Epistemic Uncertainty in Predictions for Safer Health AI Performance Under Dataset Shifts. Studies in Health Technology and Informatics. 2025. doi:10.3233/shti251493
BibTeX
@article{FernndezNarro2025,
author = {Fernández-Narro, David and Ferri, Pablo and García-Gómez, Juan Miguel and Sáez, Carlos},
title = {Quantifying Epistemic Uncertainty in Predictions for Safer Health AI Performance Under Dataset Shifts},
journal = {Studies in Health Technology and Informatics},
year = {2025},
doi = {10.3233/shti251493}
}
EUVIMED