EUVIMEDOpdag EUVIMED
ValidierungsstudieOpen access

Detecting, Characterizing, and Mitigating Implicit and Explicit Racial Biases in Health Care Datasets With Subgroup Learnability: Algorithm Development and Validation Study

Faris Gulamali, Ashwin Shreekant Sawant, Lora Liharska, Carol Horowitz, Lili Chan, Ira Hofer, Karandeep Singh, Lynne Richardson, Emmanuel Mensah, Alexander Charney, David Reich, Jianying Hu and 1 more

Journal of Medical Internet Research · 2025

DOI: 10.2196/71757 ↗

At a glance

Validierungsstudie in Journal of Medical Internet Research, 2025. Written by Faris Gulamali and 12 co-authors. Institution: Icahn School of Medicine at Mount Sinai, 1468 Madison Avenue, New York, NY, 10029, United States, 1 2122416500. Major MEDLINE topics: Bias, Routinely Collected Health Data, Race Factors, Machine Learning. Cited 5 times so far (Crossref); 58 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

The growing adoption of diagnostic and prognostic algorithms in health care has led to concerns about the perpetuation of algorithmic bias against disadvantaged groups of individuals. Deep learning methods to detect and mitigate bias have revolved around modifying models, optimization strategies, and threshold calibration with varying levels of success and tradeoffs. However, there have been limited substantive efforts to address bias at the level of the data used to generate algorithms in health care datasets.

Objective

The aim of this study is to create a simple metric (AEquity) that uses a learning curve approximation to distinguish and mitigate bias via guided dataset collection or relabeling.

Methods

We demonstrate this metric in 2 well-known examples, chest X-rays and health care cost utilization, and detect novel biases in the National Health and Nutrition Examination Survey.

Results

We demonstrated that using AEquity to guide data-centric collection for each diagnostic finding in the chest radiograph dataset decreased bias by between 29% and 96.5% when measured by differences in area under the curve. Next, we wanted to examine (1) whether AEquity worked on intersectional populations and (2) if AEquity is invariant to different types of fairness metrics, not just area under the curve. Subsequently, we examined the effect of AEquity on mitigating bias when measured by false negative rate, precision, and false discovery rate for Black patients on Medicaid. When we examined Black patients on Medicaid, at the intersection of race and socioeconomic status, we found that AEquity-based interventions reduced bias across a number of different fairness metrics including overall false negative rate by 33.3% (bias reduction absolute=1.88×10 −1 , 95% CI 1.4×10 −1 to 2.5×10 −1 ; bias reduction of 33.3%, 95% CI 26.6%‐40%; precision bias by 7.50×10 −2 , 95% CI 7.48×10 −2 to 7.51×10 −2 ; bias reduction of 94.6%, 95% CI 94.5%‐94.7%; false discovery rate by 94.5%; absolute bias reduction=3.50×10 −2 , 95% CI 3.49×10 −2 to 3.50×10 −2 ). Similarly, AEquity-guided data collection demonstrated bias reduction of up to 80% on mortality prediction with the National Health and Nutrition Examination Survey (bias reduction absolute=0.08, 95% CI 0.07-0.09). Then, we wanted to compare AEquity to state-of-the-art data-guided debiasing measures such as balanced empirical risk minimization and calibration. Consequently, we benchmarked against balanced empirical risk minimization and calibration and showed that AEquity-guided data collection outperforms both standard approaches. Moreover, we demonstrated that AEquity works on fully connected networks; convolutional neural networks such as ResNet-50; transformer architectures such as VIT-B-16, a vision transformer with 86 million parameters; and nonparametric methods such as Light Gradient-Boosting Machine.

Conclusions

In short, we demonstrated that AEquity is a robust tool by applying it to different datasets, algorithms, and intersectional analyses and measuring its effectiveness with respect to a range of traditional fairness metrics.

Subjects

MeSH terms (MEDLINE)

BiasRoutinely Collected Health DataRace FactorsMachine LearningHumansDatasets as TopicRadiography, ThoracicHealth Care CostsBlack or African AmericanMedicaidSocial ClassNutrition SurveysData CurationMortalityUnited States

Author keywords

biasdata-centric artificial intelligencefairnessmachine learning

Authors and institutions

  • Icahn School of Medicine at Mount Sinai, 1468 Madison Avenue, New York, NY, 10029, United States, 1 2122416500.
  • University of California, San Diego, San Diego, CA, United States.
  • Christiana Care Health System, Wilmington, DE, United States.
  • IBM Research, Yorktown Heights, NY, United States.

Cite

APA

Gulamali, F., Sawant, A. S., Liharska, L., Horowitz, C., Chan, L., Hofer, I., Singh, K., Richardson, L., Mensah, E., Charney, A., Reich, D., Hu, J., & Nadkarni, G. (2025). Detecting, Characterizing, and Mitigating Implicit and Explicit Racial Biases in Health Care Datasets With Subgroup Learnability: Algorithm Development and Validation Study. Journal of Medical Internet Research. https://doi.org/10.2196/71757

Vancouver

Gulamali F, Sawant AS, Liharska L, Horowitz C, Chan L, Hofer I, et al. Detecting, Characterizing, and Mitigating Implicit and Explicit Racial Biases in Health Care Datasets With Subgroup Learnability: Algorithm Development and Validation Study. Journal of Medical Internet Research. 2025. doi:10.2196/71757

BibTeX
@article{Gulamali2025,
  author = {Gulamali, Faris and Sawant, Ashwin Shreekant and Liharska, Lora and Horowitz, Carol and Chan, Lili and Hofer, Ira and Singh, Karandeep and Richardson, Lynne and Mensah, Emmanuel and Charney, Alexander and Reich, David and Hu, Jianying and Nadkarni, Girish},
  title = {Detecting, Characterizing, and Mitigating Implicit and Explicit Racial Biases in Health Care Datasets With Subgroup Learnability: Algorithm Development and Validation Study},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  doi = {10.2196/71757}
}
Start for free →