Loss function influence on hyperparameter optimization for observational healthcare prediction models
Journal of the American Medical Informatics Association · 01.08.2026 · 33 (8) · 1474-1484
DOI: 10.1093/jamia/ocag075 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 14.05.2026. Verfasst von Fleur Vereijken und 3 weiteren Autor:innen. Beteiligte Einrichtung: Erasmus University Medical Center, Rotterdam, The Netherlands. Hauptthemen laut MEDLINE: Machine Learning. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 44 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Johnson & Johnson.
Abstract (Originalsprache)
Objectives
Prediction models are increasingly used in healthcare for risk stratification and personalized care. Many models are developed using machine learning, which requires tuning hyperparameters to maximize performance based on a chosen loss function metric. In healthcare, the area under the receiver operating characteristic curve (AUROC) is commonly used for this purpose, but it may not always be the most appropriate choice for every clinical application. We empirically characterize whether the choice of loss function metric in hyperparameter optimization leads to systematic differences in model behavior across several clinical prediction tasks using real-world healthcare data.
Methods
We utilized fifteen different loss function metrics to guide hyperparameter selection across three clinical prediction tasks and four machine learning algorithms. We then compared how loss function metric choice affected selected hyperparameters, overall performance, and individual predicted probabilities.
Results
We observed that certain hyperparameters tended to have similar optimal values across different loss function metrics, although this pattern differed by algorithm. The best-performing models, evaluated using AUROC, were often not the models with hyperparameters optimized using AUROC. While models performed similarly at a population level, based on discrimination and calibration. The choice of the loss function metric had significant impact on the individual predicted risk for a patient.
Discussion
The predictive multiplicity observed can have significant impact on the patient level, while not observed in the population level model evaluation.
Conclusion
Predictive multiplicity can have a serious impact on patient treatment decisions but is not yet well understood.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Erasmus University Medical Center, Rotterdam, The Netherlands.
- Janssen Research and Development, Titusville, NJ, United States.
ORCID-Profile: Fleur Vereijken, Peter Rijnbeek, Ross D Williams
Förderung
- Johnson & Johnson
Zitieren
APA
Vereijken, F., Reps, J. M., Rijnbeek, P., & Williams, R. D. (2026). Loss function influence on hyperparameter optimization for observational healthcare prediction models. Journal of the American Medical Informatics Association, 33(8), 1474-1484. https://doi.org/10.1093/jamia/ocag075
Vancouver
Vereijken F, Reps JM, Rijnbeek P, Williams RD. Loss function influence on hyperparameter optimization for observational healthcare prediction models. Journal of the American Medical Informatics Association. 2026;33(8):1474-1484. doi:10.1093/jamia/ocag075
BibTeX
@article{Vereijken2026,
author = {Vereijken, Fleur and Reps, Jenna M and Rijnbeek, Peter and Williams, Ross D},
title = {Loss function influence on hyperparameter optimization for observational healthcare prediction models},
journal = {Journal of the American Medical Informatics Association},
year = {2026},
volume = {33},
number = {8},
pages = {1474--1484},
doi = {10.1093/jamia/ocag075}
}
EUVIMED