EUVIMEDSkopri EUVIMED
Open access

Methods to Adjust for Confounding in Test-Negative Design COVID-19 Effectiveness Studies: Simulation Study

Elizabeth Ak Rowley iD, Patrick K Mitchell iD, Duck-Hye Yang iD, Ned Lewis iD, Brian E Dixon iD, Gabriela Vazquez-Benitez iD, William F Fadel iD, Inih J Essien iD, Allison L Naleway iD, Edward Stenehjem iD, Toan C Ong iD, Manjusha Gaglani iD and 6 more

JMIR Formative Research · 2025

DOI: 10.2196/58981 ↗

At a glance

Journal article in JMIR Formative Research, 2025. Written by Elizabeth Ak Rowley and 17 co-authors. Institution: Westat, Rockville, MD, United States. Major MEDLINE topics: COVID-19, COVID-19 Vaccines, Vaccine Efficacy, Research Design. Cited 2 times so far (Crossref); 48 references. The full text is freely available in PubMed Central.

AstrattSubjectsDettalji biblijografiċiCiteRelated studies

Abstract (original language)

Background

Real-world COVID-19 vaccine effectiveness (VE) studies are investigating exposures of increasing complexity accounting for time since vaccination. These studies require methods that adjust for the confounding that arises when morbidities and demographics are associated with vaccination and the risk of outcome events. Methods based on propensity scores (PS) are well-suited to this when the exposure is dichotomous, but present challenges when the exposure is multinomial.

Objective

This simulation study aimed to investigate alternative methods to adjust for confounding in VE studies that have a test-negative design.

Methods

Adjustment for a disease risk score (DRS) is compared with multivariable logistic regression. Both stratification on the DRS and direct covariate adjustment of the DRS are examined. Multivariable logistic regression with all the covariates and with a limited subset of key covariates is considered. The performance of VE estimators is evaluated across a multinomial vaccination exposure in simulated datasets.

Results

Bias in VE estimates from multivariable models ranged from –5.3% to 6.1% across 4 levels of vaccination. Standard errors of VE estimates were unbiased, and 95% coverage probabilities were attained in most scenarios. The lowest coverage in the multivariable scenarios was 93.7% (95% CI 92.2%-95.2%) and occurred in the multivariable model with key covariates, while the highest coverage in the multivariable scenarios was 95.3% (95% CI 94.0%-96.6%) and occurred in the multivariable model with all covariates. Bias in VE estimates from DRS-adjusted models was low, ranging from –2.2% to 4.2%. However, the DRS-adjusted models underestimated the standard errors of VE estimates, with coverage sometimes below the 95% level. The lowest coverage in the DRS scenarios was 87.8% (95% CI 85.8%-89.8%) and occurred in the direct adjustment for the DRS model. The highest coverage in the DRS scenarios was 94.8% (95% CI 93.4%-96.2%) and occurred in the model that stratified on DRS. Although variation in the performance of VE estimates occurred across modeling strategies, variation in performance was also present across exposure groups.

Conclusions

Overall, models using a DRS to adjust for confounding performed adequately but not as well as the multivariable models that adjusted for covariates individually.

Subjects

MeSH terms (MEDLINE)

COVID-19COVID-19 VaccinesVaccine EfficacyResearch DesignHumansConfounding Factors, EpidemiologicSARS-CoV-2Computer SimulationLogistic ModelsPropensity Score

Author keywords

COVID-19assessmentcomorbiditydisease risk scorepropensity scoresimulation studyusefulnessvaccine effectiveness

Substances

COVID-19 Vaccines

Authors and institutions

  • Westat, Rockville, MD, United States.
  • Vaccine Study Center, Northern California Division of Research, Kaiser Permanente, Oakland, CA, United States.
  • Center for Biomedical Informatics, Regenstrief Institute, Indianapolis, IN, United States.
  • Fairbanks School of Public Health, Indiana University, Indianapolis, IN, United States.
  • HealthPartners Institute, Minneapolis, MN, United States.
  • Center for Health Research, Kaiser Permanente, Portland, OR, United States.

ORCID profiles: Elizabeth Ak Rowley, Patrick K Mitchell, Duck-Hye Yang, Ned Lewis, Brian E Dixon, Gabriela Vazquez-Benitez, William F Fadel, Inih J Essien, Allison L Naleway, Edward Stenehjem, Toan C Ong, Manjusha Gaglani, Karthik Natarajan, Peter Embi, Ryan E Wiegand, Ruth Link-Gelles, Mark W Tenforde, Bruce Fireman

Cite

APA

Rowley, E. A., Mitchell, P. K., Yang, D. H., Lewis, N., Dixon, B. E., Vazquez-Benitez, G., Fadel, W. F., Essien, I. J., Naleway, A. L., Stenehjem, E., Ong, T. C., Gaglani, M., Natarajan, K., Embi, P., Wiegand, R. E., Link-Gelles, R., Tenforde, M. W., & Fireman, B. (2025). Methods to Adjust for Confounding in Test-Negative Design COVID-19 Effectiveness Studies: Simulation Study. JMIR Formative Research. https://doi.org/10.2196/58981

Vancouver

Rowley EA, Mitchell PK, Yang DH, Lewis N, Dixon BE, Vazquez-Benitez G, et al. Methods to Adjust for Confounding in Test-Negative Design COVID-19 Effectiveness Studies: Simulation Study. JMIR Formative Research. 2025. doi:10.2196/58981

BibTeX
@article{Rowley2025,
  author = {Rowley, Elizabeth Ak and Mitchell, Patrick K and Yang, Duck-Hye and Lewis, Ned and Dixon, Brian E and Vazquez-Benitez, Gabriela and Fadel, William F and Essien, Inih J and Naleway, Allison L and Stenehjem, Edward and Ong, Toan C and Gaglani, Manjusha and Natarajan, Karthik and Embi, Peter and Wiegand, Ryan E and Link-Gelles, Ruth and Tenforde, Mark W and Fireman, Bruce},
  title = {Methods to Adjust for Confounding in Test-Negative Design COVID-19 Effectiveness Studies: Simulation Study},
  journal = {JMIR Formative Research},
  year = {2025},
  doi = {10.2196/58981}
}
Start for free →