Anytime-valid off-policy Inference for Contextual Bandits
ACM / IMS Journal of Data Science · 2024
DOI: 10.1145/3643693 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM / IMS Journal of Data Science, 2024. Verfasst von Ian Waudby-Smith und 4 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 65 Literaturangaben. Gefördert von: NSF, ARL IoBT REIGN, DEVCOM Army Research Laboratory under Cooperative Agreement.
Abstract (Originalsprache)
Contextual bandit algorithms are ubiquitous tools for active sequential experimentation in healthcare and the tech industry. They involve online learning algorithms that adaptively learn policies over time to map observed contexts X t to actions A t in an attempt to maximize stochastic rewards R t . This adaptivity raises interesting but hard statistical inference questions, especially counterfactual ones: for example, it is often of interest to estimate the properties of a hypothetical policy that is different from the logging policy that was used to collect the data—a problem known as “off-policy evaluation” (OPE). Using modern martingale techniques, we present a comprehensive framework for OPE inference that relaxes unnecessary conditions made in some past works (such as performing inference at prespecified sample sizes, uniformly bounded importance weights, constant logging policies, and constant policy values, among others), significantly improving on them both theoretically and empirically. Importantly, our methods can be employed while the original experiment is still running (that is, not necessarily post hoc), when the logging policy may be itself changing (due to learning), and even if the context distributions are a highly dependent time series (such as if they are drifting over time). More concretely, we derive confidence sequences for various functionals of interest in OPE. These include doubly robust ones for time-varying off-policy mean reward values, but also confidence bands for the entire cumulative distribution function of the off-policy reward distribution. All of our methods (a) are valid at arbitrary stopping times; (b) only make nonparametric assumptions; (c) do not require importance weights to be uniformly bounded, and if they are, we do not need to know these bounds; and (d) adapt to the empirical variance of our estimators. In summary, our methods enable anytime-valid off-policy inference using adaptively collected contextual bandit data.
Förderung
- NSF
- ARL IoBT REIGN
- DEVCOM Army Research Laboratory under Cooperative Agreement
Zitieren
APA
Waudby-Smith, I., Wu, L., Ramdas, A., Karampatziakis, N., & Mineiro, P. (2024). Anytime-valid off-policy Inference for Contextual Bandits. ACM / IMS Journal of Data Science. https://doi.org/10.1145/3643693
Vancouver
Waudby-Smith I, Wu L, Ramdas A, Karampatziakis N, Mineiro P. Anytime-valid off-policy Inference for Contextual Bandits. ACM / IMS Journal of Data Science. 2024. doi:10.1145/3643693
BibTeX
@article{WaudbySmith2024,
author = {Waudby-Smith, Ian and Wu, Lili and Ramdas, Aaditya and Karampatziakis, Nikos and Mineiro, Paul},
title = {Anytime-valid off-policy Inference for Contextual Bandits},
journal = {ACM / IMS Journal of Data Science},
year = {2024},
doi = {10.1145/3643693}
}
EUVIMED