Evaluating large-scale propensity score adjustment when sample size is small
Journal of the American Medical Informatics Association · 2026
DOI: 10.1093/jamia/ocag133 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, 2026. Verfasst von Fleur Vereijken und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, 3015 GD, The Netherlands. Hauptthemen laut MEDLINE: Propensity Score, Observational Studies as Topic, Models, Statistical. Die Arbeit nennt 26 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Johnson and Johnson, NIH HHS, VA.
Abstract (Originalsprache)
Background
Large-scale propensity score (LSPS) models are increasingly used to control confounding in observational studies, but their reliability in small-sample settings is unclear. Small samples can limit a study’s ability to estimate propensity scores accurately and achieve adequate covariate balance, raising concerns about insufficient confounding adjustment. Resulting in small datasets being excluded, despite potentially containing valid information.
Methods
We evaluated the balancing and bias reduction performance of LSPS models using six target-comparator pairs. To emulate a distributed data network analysis setting, we partitioned two large real-world data sources into smaller subsets. Within each subset, LSPS models were fit locally and evaluated on covariate balance, treatment effect bias and precision after empirical calibration, using real negative controls and synthetic positive controls. Performance was compared to a global LSPS model trained on the full dataset.
Results
Across most target-comparator pairs, LSPS models substantially reduced bias and, when applying empirical calibration, improved precision. PS matching was more resilient to small sample sizes than PS stratification, achieving adequate balance even at n = 500, while stratification sometimes failed at n = 4000. A slight bias increase was observed at the smallest sample sizes, though not universally. Standard balance diagnostics consistently failed below n = 20 000, while a recently proposed diagnostic accounting for chance imbalance did not.
Conclusions
LSPS models generally provide reliable bias reduction in small-sample settings, supporting their use in federated analyses. However, standard balance diagnostics may be misleading in small samples, and alternatives should be considered, such as significance checking. When LSPS fails to reduce bias adequately, additional adjustment strategies are required.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, 3015 GD, The Netherlands.
- Observational Health Data Analytics, Janssen Research and Development, Titusville, NJ, 08560, United States.
- Department of Biostatistics, University of California, Los Angeles, Los Angeles, CA 90095-1772, United States.
- Data Science and Biostatistics, Washington University School of Medicine, Saint Louis, Missouri, MO 63110, United States.
- Department of Biomedical Informatics, Columbia University Medical Center, New York, NY 10032, United States.
ORCID-Profile: Fleur Vereijken, Marc A Suchard, Linying Zhang, Peter R Rijnbeek, Ross D Williams, Martijn J Schuemie
Förderung
- Johnson and Johnson
- NIH HHS
- VA
- Janssen Research & Development and Gilead Sciences
- National Institutes of Health
- Department of Veterans Affairs
Zitieren
APA
Vereijken, F., Reps, J. M., Suchard, M. A., Zhang, L., Hripcsak, G., Rijnbeek, P. R., Williams, R. D., & Schuemie, M. J. (2026). Evaluating large-scale propensity score adjustment when sample size is small. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocag133
Vancouver
Vereijken F, Reps JM, Suchard MA, Zhang L, Hripcsak G, Rijnbeek PR, et al. Evaluating large-scale propensity score adjustment when sample size is small. Journal of the American Medical Informatics Association. 2026. doi:10.1093/jamia/ocag133
BibTeX
@article{Vereijken2026,
author = {Vereijken, Fleur and Reps, Jenna M and Suchard, Marc A and Zhang, Linying and Hripcsak, George and Rijnbeek, Peter R and Williams, Ross D and Schuemie, Martijn J},
title = {Evaluating large-scale propensity score adjustment when sample size is small},
journal = {Journal of the American Medical Informatics Association},
year = {2026},
doi = {10.1093/jamia/ocag133}
}
EUVIMED