EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Sampling for computational efficiency when conducting analyses in big data

Jacqueline E Rudolph iD, Yiyi Zhou, Karine Yenokyan, Xiaoqiang Xu, Eryka Wentz, Keri L Calkins, Corinne E Joshu, Bryan Lau iD

American Journal of Epidemiology · 2025

DOI: 10.1093/aje/kwaf268 ↗

Kurz gefasst

Fachartikel in der Zeitschrift American Journal of Epidemiology, 2025. Verfasst von Jacqueline E Rudolph und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Epidemiology, Bloomberg School of Public Health, Johns Hopkins University, Baltimore, MD, United States. Hauptthemen laut MEDLINE: Lung Neoplasms, Big Data. Die Arbeit nennt 18 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NIAAA NIH HHS, NIAID NIH HHS, NCI NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

A challenge to research in big data is the inherent computational intensity of analyses, particularly when using rigorous methods to address biases. We demonstrate the use of sampling methods in big data to estimate parameters using fewer resources. Our motivating question was whether lung cancer incidence differs by baseline HIV status, using a cohort of nearly 30 million Medicaid beneficiaries. We targeted three parameters (with listed estimator): incidence rate ratio (IRR, Poisson model), HR (Cox model), and risk ratio (RR, Kaplan–Meier). We controlled for confounders using inverse probability weighting. We ran analyses using the full sample and several sampling schemes: divide-and-recombine (10, 20, 50 samples), subcohort, and case-cohort. We compared point estimates, standard errors, computation time, and memory used. We observed 1113 incident lung cancer diagnoses among 180 980 beneficiaries with HIV and 33 106 diagnoses among 29 179 940 beneficiaries without HIV. Findings were similar across target parameters. The subcohort and case-cohort approaches had estimates closer to the full sample and were faster and less memory intensive than divide-and-recombine, especially when estimating the risk ratio. Including nonsampled cases in the case-cohort resulted in increases in computation time and memory relative to the subcohort approach.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Lung NeoplasmsBig DataHumansHIV InfectionsUnited StatesFemaleMaleIncidenceProportional Hazards ModelsMiddle AgedMedicaidSampling StudiesPoisson Distribution

Stichwörter der Autor:innen

big datacase-cohortcomputational efficiencydivide-and-recombinesampling methodssubcohort

Autor:innen und Institutionen

  • Department of Epidemiology, Bloomberg School of Public Health, Johns Hopkins University, Baltimore, MD, United States.
  • Department of Medicine, School of Medicine, Johns Hopkins University, Baltimore, MD, United States.
  • Mathematica, Ann Arbor, MI, United States.

ORCID-Profile: Jacqueline E Rudolph, Bryan Lau

Förderung

  • NIAAA NIH HHS
  • NIAID NIH HHS
  • NCI NIH HHS
  • Johns Hopkins University Center
  • NIH HHS
  • National Institutes of Health

Zitieren

APA

Rudolph, J. E., Zhou, Y., Yenokyan, K., Xu, X., Wentz, E., Calkins, K. L., Joshu, C. E., & Lau, B. (2025). Sampling for computational efficiency when conducting analyses in big data. American Journal of Epidemiology. https://doi.org/10.1093/aje/kwaf268

Vancouver

Rudolph JE, Zhou Y, Yenokyan K, Xu X, Wentz E, Calkins KL, et al. Sampling for computational efficiency when conducting analyses in big data. American Journal of Epidemiology. 2025. doi:10.1093/aje/kwaf268

BibTeX
@article{Rudolph2025,
  author = {Rudolph, Jacqueline E and Zhou, Yiyi and Yenokyan, Karine and Xu, Xiaoqiang and Wentz, Eryka and Calkins, Keri L and Joshu, Corinne E and Lau, Bryan},
  title = {Sampling for computational efficiency when conducting analyses in big data},
  journal = {American Journal of Epidemiology},
  year = {2025},
  doi = {10.1093/aje/kwaf268}
}
Kostenlos starten →