EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Loss function influence on hyperparameter optimization for observational healthcare prediction models

Fleur Vereijken iD, Jenna M Reps, Peter Rijnbeek iD, Ross D Williams iD

Journal of the American Medical Informatics Association · 01.08.2026 · 33 (8) · 1474-1484

DOI: 10.1093/jamia/ocag075 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 14.05.2026. Verfasst von Fleur Vereijken und 3 weiteren Autor:innen. Beteiligte Einrichtung: Erasmus University Medical Center, Rotterdam, The Netherlands. Hauptthemen laut MEDLINE: Machine Learning. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 44 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Johnson & Johnson.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

Prediction models are increasingly used in healthcare for risk stratification and personalized care. Many models are developed using machine learning, which requires tuning hyperparameters to maximize performance based on a chosen loss function metric. In healthcare, the area under the receiver operating characteristic curve (AUROC) is commonly used for this purpose, but it may not always be the most appropriate choice for every clinical application. We empirically characterize whether the choice of loss function metric in hyperparameter optimization leads to systematic differences in model behavior across several clinical prediction tasks using real-world healthcare data.

Methods

We utilized fifteen different loss function metrics to guide hyperparameter selection across three clinical prediction tasks and four machine learning algorithms. We then compared how loss function metric choice affected selected hyperparameters, overall performance, and individual predicted probabilities.

Results

We observed that certain hyperparameters tended to have similar optimal values across different loss function metrics, although this pattern differed by algorithm. The best-performing models, evaluated using AUROC, were often not the models with hyperparameters optimized using AUROC. While models performed similarly at a population level, based on discrimination and calibration. The choice of the loss function metric had significant impact on the individual predicted risk for a patient.

Discussion

The predictive multiplicity observed can have significant impact on the patient level, while not observed in the population level model evaluation.

Conclusion

Predictive multiplicity can have a serious impact on patient treatment decisions but is not yet well understood.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Machine LearningPredictive Learning ModelsHumansPrediction AlgorithmsROC CurveArea Under CurveAlgorithms

Stichwörter der Autor:innen

Area Under the Curve (AUC)evaluation metricshyperparameter tuningmachine learningpredictive modelling

Autor:innen und Institutionen

  • Erasmus University Medical Center, Rotterdam, The Netherlands.
  • Janssen Research and Development, Titusville, NJ, United States.

ORCID-Profile: Fleur Vereijken, Peter Rijnbeek, Ross D Williams

Förderung

  • Johnson & Johnson

Zitieren

APA

Vereijken, F., Reps, J. M., Rijnbeek, P., & Williams, R. D. (2026). Loss function influence on hyperparameter optimization for observational healthcare prediction models. Journal of the American Medical Informatics Association, 33(8), 1474-1484. https://doi.org/10.1093/jamia/ocag075

Vancouver

Vereijken F, Reps JM, Rijnbeek P, Williams RD. Loss function influence on hyperparameter optimization for observational healthcare prediction models. Journal of the American Medical Informatics Association. 2026;33(8):1474-1484. doi:10.1093/jamia/ocag075

BibTeX
@article{Vereijken2026,
  author = {Vereijken, Fleur and Reps, Jenna M and Rijnbeek, Peter and Williams, Ross D},
  title = {Loss function influence on hyperparameter optimization for observational healthcare prediction models},
  journal = {Journal of the American Medical Informatics Association},
  year = {2026},
  volume = {33},
  number = {8},
  pages = {1474--1484},
  doi = {10.1093/jamia/ocag075}
}
Kostenlos starten →