EUVIMEDDescoperă EUVIMED
Open access

Assessing Covariate Balance With Small Sample Sizes

George Hripcsak iD, Linying Zhang, Yong Chen iD, Kelly Li, Marc A Suchard iD, Patrick B Ryan, Martijn J Schuemie iD

Statistics in Medicine · 2025

DOI: 10.1002/sim.70212 ↗

At a glance

Journal article in Statistics in Medicine, 2025. Written by George Hripcsak and 6 co-authors. Institution: Department of Biomedical Informatics, Columbia University Irving Medical Center, New York, New York, USA. Major MEDLINE topics: Propensity Score. Cited 17 times so far (Crossref); 51 references. The full text is freely available in PubMed Central. Funded by: NIH HHS, NLM NIH HHS, NHLBI NIH HHS.

RezumatSubjectsDate bibliograficeCiteRelated studies

Abstract (original language)

Propensity score adjustment addresses confounding by balancing covariates in subject treatment groups through matching, stratification, or weighting. Diagnostics test the success of adjustment. For example, if the standardized mean difference (SMD) for a relevant covariate exceeds a threshold like 0.1, the covariate is considered imbalanced and the study may be biased. Unfortunately, for studies with small or moderate numbers of subjects, the probability of identifying a study as biased because of chance imbalance can be grossly larger than a given nominal level like 0.05, yet that chance imbalance may not cause significant bias. In this paper, we illustrate that chance imbalance is operative in real‐world settings even for moderate sample sizes of 2000. We identify a previously unrecognized challenge that as meta‐analyses increase the precision of an effect estimate, the diagnostics must also undergo meta‐analysis for a corresponding increase in precision. We propose an alternative diagnostic that checks whether the SMD statistically significantly exceeds the threshold. Through simulation and real‐world data, we find that this diagnostic achieves a better trade‐off of type 1 error rate and power than standard nominal threshold tests and not testing for sample sizes from 250 to 4000 and for 20 to 100 000 covariates. We confirm that in network studies, meta‐analysis of effect estimates must be accompanied by meta‐analysis of the diagnostics or else systematic confounding may overwhelm the estimated effect. Our procedure supports the review of large numbers of covariates, enabling more rigorous diagnostics.

Subjects

MeSH terms (MEDLINE)

Propensity ScoreHumansSample SizeComputer SimulationBiasMeta-Analysis as TopicData Interpretation, StatisticalModels, Statistical

Author keywords

confoundingcovariate balancemeta‐analysisobservational researchpropensity score

Authors and institutions

  • Department of Biomedical Informatics, Columbia University Irving Medical Center, New York, New York, USA.
  • Observational Health Data Science and Informatics, New York, New York, USA.
  • VA Informatics and Computing Infrastructure, US Department of Veterans Affairs, Salt Lake City, Utah, USA.
  • Institute for Informatics, Data Science and Biostatistics, Washington University in St. Louis, St. Louis, Missouri, USA.
  • The Center for Health AI and Synthesis of Evidence (CHASE), University of Pennsylvania, Philadelphia, Pennsylvania, USA.
  • Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania, Philadelphia, Pennsylvania, USA.

ORCID profiles: George Hripcsak, Yong Chen, Marc A Suchard, Martijn J Schuemie

Funding

  • NIH HHS
  • NLM NIH HHS
  • NHLBI NIH HHS
  • National Institutes of Health

Cite

APA

Hripcsak, G., Zhang, L., Chen, Y., Li, K., Suchard, M. A., Ryan, P. B., & Schuemie, M. J. (2025). Assessing Covariate Balance With Small Sample Sizes. Statistics in Medicine. https://doi.org/10.1002/sim.70212

Vancouver

Hripcsak G, Zhang L, Chen Y, Li K, Suchard MA, Ryan PB, et al. Assessing Covariate Balance With Small Sample Sizes. Statistics in Medicine. 2025. doi:10.1002/sim.70212

BibTeX
@article{Hripcsak2025,
  author = {Hripcsak, George and Zhang, Linying and Chen, Yong and Li, Kelly and Suchard, Marc A and Ryan, Patrick B and Schuemie, Martijn J},
  title = {Assessing Covariate Balance With Small Sample Sizes},
  journal = {Statistics in Medicine},
  year = {2025},
  doi = {10.1002/sim.70212}
}
Start for free →