A Simplified Metric to Streamline Between-Group Fairness Assessment for Predictive Models: Algorithm Development and Evaluation Study
JMIR AI · 2026-08-24 · 5 · e85822-e85822
DOI: 10.2196/85822 ↗At a glance
Journal article in JMIR AI, published online on 2026-08-24. Written by Haoyuan Wang and 3 co-authors. Institution: Department of Biostatistics and Bioinformatics, Duke University School of Medicine, 2424 Erwin Rd, Durham, NC, 27705, United States, 1 919-613-3665. 19 references. The full text is freely available in PubMed Central. Funded by: NIMH NIH HHS.
Abstract (original language)
Background
Fairness evaluation is essential for trustworthy clinical risk prediction. However, existing fairness-oriented discrimination metrics either ignore cross-group comparisons or rely on exhaustive pairwise evaluations, making them difficult to interpret and impractical for model selection.
Objective
This study aimed to develop and evaluate novel fairness-oriented discrimination metrics for clinical risk prediction that address limitations of within-group and pairwise cross-group approaches.
Methods
We examined theoretical properties of existing U-statistic–based metrics, including concordance index (CI) and area under the receiver operating characteristic curve (AUC), when applied to subgroups. We highlighted the distinction between within-group discrimination (ranking within a subgroup) and group-level discrimination (ranking relative to the broader population). Building on this framework, we proposed group-level extensions of the CI and AUC that summarize subgroup-specific performance in a single interpretable measure. We then applied these metrics to the PREVENT (Predicting Risk of Cardiovascular Disease Events) equation, a recently developed model for atherosclerotic cardiovascular disease.
Results
The traditional subgroup-specific CI and AUC captured within-group but not group-level discrimination, obscuring inequities in clinical decision-making. Existing cross-group approaches (eg, the xCI and xAUC metrics) addressed this limitation but became computationally and interpretively burdensome with multiple subgroups due to pairwise comparisons. Our proposed metrics provided a streamlined alternative, yielding 1 summary statistic per subgroup while retaining sensitivity to cross-group ranking disparities. Applied to PREVENT, these metrics revealed differences in subgroup performance not apparent from within-group evaluations.
Conclusions
By distinguishing between within-group and group-level discrimination, our framework clarifies a common source of misinterpretation in fairness evaluation. The proposed group-level extensions of the CI and AUC provide practical, interpretable tools for evaluating fairness in clinical prediction models, enabling more transparent and equitable risk assessment.
Subjects
Author keywords
Authors and institutions
- Department of Biostatistics and Bioinformatics, Duke University School of Medicine, 2424 Erwin Rd, Durham, NC, 27705, United States, 1 919-613-3665.
Funding
- NIMH NIH HHS
Cite
APA
Wang, H., Hong, C., Pencina, M., & Engelhard, M. (2026). A Simplified Metric to Streamline Between-Group Fairness Assessment for Predictive Models: Algorithm Development and Evaluation Study. JMIR AI, 5, e85822-e85822. https://doi.org/10.2196/85822
Vancouver
Wang H, Hong C, Pencina M, Engelhard M. A Simplified Metric to Streamline Between-Group Fairness Assessment for Predictive Models: Algorithm Development and Evaluation Study. JMIR AI. 2026;5:e85822-e85822. doi:10.2196/85822
BibTeX
@article{Wang2026,
author = {Wang, Haoyuan and Hong, Chuan and Pencina, Michael and Engelhard, Matthew},
title = {A Simplified Metric to Streamline Between-Group Fairness Assessment for Predictive Models: Algorithm Development and Evaluation Study},
journal = {JMIR AI},
year = {2026},
volume = {5},
pages = {e85822--e85822},
doi = {10.2196/85822}
}
EUVIMED